All,
I am still running across a number of calibrations that @gcalkins flagged as questionable and I don’t recall seeing better explanations. Maybe I just missed them. Maybe it will take a while to get to them all? I was just wondering because it can get frustrating at times, but I realize that the team must be extremely busy. Thank you all for your work on this very important research.
Dear @annettei,
Thank you for letting us know about this. We are taking yours and @gcalkins observations and concerns very seriously. I have assembled a new, highly specialized team to address this issue, who will be monitoring and correcting vessel issues on a regular and frequent schedule beginning tomorrow.
Thank you for your kind patience. We are indeed quite busy, but this is also very important both to the experience of volunteers such as yourself, and ultimately to the research results. We value this feedback highly and hope to hear from you and others about whether or not your experience with both movie quality and feedback quality improves over the next days and weeks.
Best wishes,
Pietro
Thank you @pietro! I can honestly say overall this has been a very satisfying experience. I appreciate your efforts in making this an even better experience and helping us become more accurate.
By the way, there is no need to respond to me on weekends. It can wait until the work week so you can enjoy your weekends 
Annette
Dear @pietro . (cc: @mh973 , @lvinarcsik , @seplute )
Great to hear the team is back on the case to get these questionable calibrations cleaned up, changed, removed, or edited with better expert explanations.
Input for the team:
1.) I have not seen questionable calibrations #2, #3, #4, #5, #9 or #11 in quite some time, These may have already been removed. It was quite noticeable and a great improvement not to see these in the dataset over the past 6-8 weeks. Thank you.
2.) Questionable calibrations #6, #7, #8, and #12 were recently discussed above. Still seeing these in the dataset and while advanced players know to answer “flowing” new players will likely be confused given the conflicting sight of an apparent stall in the central vessel and unclear outlining. Better explanation needed at a minimum for these. (Perhaps it would just be better to just put these in a training video to show the subtlety around what is the targeted vessel and what is not.)
3.) Questionable calibration #1 seems to have reappeared just lately and needs to be removed again.
4.) I also have identified questionable calibrations #10, #13, #14, and #15 that need attention. Some of these have been observed since the original reporting 2 months ago. I can give you details on these when you are ready.
I have added a comment to the comment fields for each of these questionable calibrations that should be string searchable, e.g. “Questionable Calibration #1”. (Or you can just look for calibration movies that have lots of comments and you’ll probably find the same ones.)
Thanks for addressing these concerns.
Re: Questionable Calibration #1
Hi @pietro (and calibrations team)
Going to give you the remaining questionable calibrations one at a time so each note doesn’t get as long and we can discuss the merits or issues with keeping or removing each calibration.
This questionable calibration was one of the first ones discovered. I thought it had been removed, but have seen it a couple more times in the last week or two.
The expert has marked this one as flowing. It appears to have several dark spots that are not moving.
The voting was Stalled 205 versus Flowing 253 a week or two ago. (Many of us have started voting flowing only to avoid being penalized.) A few notable comments
• Nozomi Nishimura2016-09-18 00:04:30
The two black dots at the lower end are not moving. These look like cells to me, so I scored it as stuck
• caprarom2016-10-05 15:37:50
Agreed
• Sean40462016-10-15 05:29:41
26-31 agreed.
• caprarom2016-10-15 13:01:16
Yep, still agreed.
• gcalkins2017-04-12 22:23:17
Looks like a stall to me. Maybe time to remove this calibration.
• LotteryDiscountz2017-04-24 14:29:36
~27-35 lower half of outline has a stall imo
• PietroM2017-04-29 20:29:43
Yes, but… if there is truly a stall, then the rest of the vessel would not appear to have a moving texture. This is a tricky one and I have flagged it for our expert to reconsider.
If there is this much confusion by advanced players, it should probably be removed.
Thanks
@gcalkins
Hi @gcalkins!
This is just a quick update to you and all Catchers that we have developed an internal process (more localized within the eyesonalz operational group) to ensure ongoing bad movie and bad feedback remediation. 50 vessels from our flagged list of bad feedback were updated yesterday (8 were corrected and the rest were clarified with detailed explanations that will appear the next time you see them). Tories, the person with the greatest expertise, who defines our ground truth, will continue to chip away at 400 of these flagged calibration movies over the next couple of weeks, and I will provide periodic updates on our clean-up progress. We also have @lvinarcsik, who is also expert in these matters and in the biomedical research, who is checking all of the “bad movies”. I am very excited and grateful to these two highly qualified members of our team who have graciously taken on these tasks, with a full appreciation of how the quality of these movies and outlines affects the experience of volunteers like you.
Thank you again for remaining steadfast on these issues. We will respond to your specific inquiries above as soon as we are able.
All best,
Pietro
Calibration movie remediation update for Wednesday, July 12, 2017:
150 more videos screened:
20 movies removed
43 movies with changes/additional comments
Many thanks to Torie for this tremendous effort.
@gcalkins and others - please let us know if you can see these improvements when you are Catching!
Best,
Pietro
Hi @pietro (and Torie and Lindsay @lvinarcsik)
A big THANK YOU to everyone involved in this effort. The improvement over the last three days has been very obvious to me. I have not come across a single questionable calibration since the removals and changes were made. I believe you will see a definite rise in the average sensitivity scores of all players (and less frustration from false penalties). I will let you know if I should come across any more in the coming days and weeks.
Thanks again and have a great weekend,
@gcalkins
Thank you for these words of encouragement @gcalkins! It means a lot to the whole team, especially coming from you.
Enjoy the weekend,
@pietro
To @pietro, Tories, and @lvinarcsik, I would also like to thank you for your efforts in reviewing questionable calibrations. I did run across a few questionable calibrations on Friday and Saturday, but they weren’t enough to decimate my sensitivity, like the situation I ran into last weekend. I understand it will take a while to get them all, and what you’ve done so far has already made participating in this effort a better experience. Thanks again.
Annette
That’s great news, @annettei! Thank you for the feedback!
Torie completed her review of ALL the calibrations on Sunday night. If you see something that doesn’t make sense, please flag it again. Torie is planning to make a second pass in a few weeks to resolve any residual issues.
We are still working on removing the vessel movies with bad outlines. Will keep you posted.
Best wishes,
Pietro
Hi @pietro - (cc: Torie and Lindsay @lvinarcsik )
I did come across a couple of our previously identified questionable calibrations. Specifically #10 and #15.
Re: Questionable Calibration #10
This one appears to be misidentified as “stalled”, when a frame by frame analysis shows the dark spot is clearly moving. This one needs to be changed to “flowing” or removed. (If you change it, please let us know once it is done.) The yellow boxes provide a fixed reference point in the image that shows the changing distance to the dark spot highlighted by the red arrows. The movie does have an unwanted “shift” in frame 32 and it also helps to look at a few additional frames beyond those marked by the expert. I have skipped a few frames to accentuate the movement and changes of the rbc’s.
Here are a few comments from other players.
• Sean40462016-11-02 14:18:33 @sean4046
I still doubt it’s a stall. Look at frames 36-38.
• caprarom2016-11-02 16:03:19 @caprarom
Agreed, Sean; it’s still uncertain.
• FlowFinder2017-04-15 15:29:03
Looks like two different capillaries running in parallel, but still think I see flow in both. Tough one.
• LotteryDiscountz2017-04-22 12:22:32 @LotteryDiscountz
I had same conclusion as FlowFinder
• caprarom2017-06-10 12:42:28 @caprarom
Yep, gc, I still keep missing this one also.
• Michael Landau2017-07-18 16:22:16 @MikeLandau
I don’t see the stall either. I really hope they get rid of this movie. I see a lot of people have already commented on this one, but I thought I would add my voice anyway for whatever it’s worth. Maybe if everyone complains about it they will finally relent, and get rid of this one.
Next one is:
Re: Questionable Calibration #15
Questionable Calibration #15 is marked as “flowing”. Perhaps the technical answer is correct that the targeted capillary does turn to the upper right near the end of the movie and may be flowing, however, the stalled segment or branch at the upper left is a very obvious artifact that is very misleading. Again, perhaps you can use this movie in a training movie, but eliminate it as a calibration movie.
Here are a few comments from players.
• pietro2016-10-02 04:13:38 @pietro
At layer 25 there appears to be a bit of vessel trying to connect from the northwest end of the green outline. That’s where it looks to me like there is a stall
• caprarom2016-10-04 23:41:18 @caprarom
I concur.
• evelynrsmith2016-12-16 01:49:56 @evelynrsmith
Now, this is third time “getting caught” on this one. There either is a Stall at 23-24 or I quit trying! Please pull this one, I never want to see this movie again! The vote for a Stall is now 90 Yes, to 84 No.
• LotteryDiscountz2017-04-23 22:48:11
A month later I concur with above where indicated
• stephencpcd2017-04-29 12:48:41
I agree, a very clear stall at 25.
• FlowFinder2017-06-21 17:34:33
Considerable confusion regarding this calibration. The issue arises from the apparent disconnect to the small protrusion in the upper left corner. The target vessel does appear to move off to the upper right eventually. Is this worth having?
Please remove these calibrations.
Thanks
@gcalkins
Re: Questionable Calibration #10
It has been a few months since we have complained about some of the questionable calibrations. Although I have finally seen this calibration enough to give the “accepted” answer, it is still wrong and continues to trip up fellow players. This calibration still needs to be removed from the system.
Here are the latest comments (Many of us have stopped commenting because they seem to be ignored by the team.) I have flagged my comment for this calibration. It should pop up in your list of flagged feedback.
Details of Questionable Calibration #10 can be seen above in the preceding post to this forum topic.
• Michael Landau2017-07-26 18:03:29
I missed this one every single time, and I just missed it again!
• Michael Landau2017-07-26 18:03:33
I missed this one every single time, and I just missed it again!
• gcalkins2017-08-05 20:29:39
Please remove this calibration. Not stalled.
• Michael Landau2017-08-07 02:57:10
I missed this one again! So frustrating!
• Christiane2017-08-08 15:08:09
Flowing at 36 ! Please remove.
• Christiane2017-08-16 18:18:35
i missed it again ! Please remove …
• Christiane2017-08-19 19:45:10
and again …
• japayt2017-08-24 16:33:29
As most of the others commented, I was focused on the earlier vessel that more properly fit the outline…
• Christiane2017-08-31 17:58:51
and i missed it for the 4th time !!! Please : remove because very frustrating
• caprarom2017-09-05 00:53:01
Well, just missed it for the second time today. Almost flagged it as a stall, but I do now agree with gc and company; it seems to be flowing.
• stephencpcd2017-10-02 17:23:21
I concur with all my colleagues, please remove, thank you.
• Michael Landau2017-10-04 16:11:29
I missed this one again, and I have to ask, how many comments does there have to be on the movie before you finally decide to remove it?
• Christiane2017-10-22 17:26:32
missed it the 5th time …
Hi guys,
Some news on this front. From the movies Torie had selected, a few were still active on the list, and #10 was one of them. Theses last movies have been removed.
#15 has a new comment: ‘Vessel is flowing. The gap you may see is not actually a stall. The two vessels in question never actually connect, they flow near each other and turn away from each other. You can confirm this as well because you can see distinct flow in each of the two segments.’ Discussing with the team what to do with that one.
We also added more expert’s comments on movies. We hope it will answer some of your questions regarding questionable calibration movies. Please continue to report them.
Dear All!
I am happy to let you know that a couple weeks ago we improved the calibration movies & their answers, with the generous help of our super catchers @caprarom & @gcalkins !!
Here’s what we did:
7 calibration movies were removed, by suggestion from the superusers & approval by the lab
21 expert comments were improved, written by @caprarom & @gcalkins, and approved by the lab!
@nn62 actually said:
1 movie answer was changed from stalled to flowing - correcting a mistake made by an expert previously, as well as our catchers, and as per explanation by @nn62.
That’s that - thanks so much @caprarom & @gcalkins again, and hope this improves everyone’s experience!
Something seems amiss with the expert comment box for the calibrations; multiple repeats of “Flowing” and “Stalled,” with the dominant response not necessarily the correct one. I’ve submitted a report form on the issue already.
)