Dealing with Questionable Calibration Movies

Stall Catchers
1

It is time to start a conversation about the 6 to 8 questionable calibration movies that have been tripped over multiple times by many of us during the competition. For the long term good of Stall Catchers and maintaining player interest you need to either remove these questionable calibrations or update the annotation with a better explanation and logic from the experts as to why they were scored as they were… “Vessel is flowing.” and “a dark spot is not moving throughout the image” are insufficient or are actually wrong for these particular 6 to 8 films.

It is frustrating to continually be reprimanded (repeatedly) for something that I thought I was scoring correctly. Learning from my mistakes is only possible if you provide a better explanation of my error. If these calibrations are actually scored incorrectly then they should be removed immediately.

Many players have left comments on these particular calibration movies. Addressing these “questionable calibrations” would be appreciated (and I think very necessary).

Thanks,
gcalkins

4 Likes
2

Dear @gcalkins,

I couldn’t agree more! In fact, you’ll find some of my own frustrated comments alongside your own. Tonight I saw a comment that was made by one of our biomedical researchers that agreed with one of your comments. So if we are aware of them, why are these flawed answers still there?

In some cases, the answers are actually correct, but we haven’t provided the necessary instruction for understanding why. For example, in rare cases, there are stalled vessels with none of the telltale “stuck stripes” that would normally suggest a stall. This can happen when a vessel movie is slightly overexposed. In these cases, the lack of textured movement along with contextual cues would lead an expert to label it as stalled. Given how rare such cases are in the experimental datasets, I think we will end up removing these, even though they are correct.

However, as you say, there are several cases where the expert answer is wrong. We fully intend to have all controversial vessels examined closely by the researchers, and indeed we are developing a researcher’s control panel that will enable them to make the corrections themselves. At present, the only way to fix these is via manual updates to our database.

Your feedback is very helpful, because as we continue to develop the Stall Catchers software we have to prioritize the feature development, which can be very tricky when there are so many improvements and new features (currently 54!) we want to add to make Stall Catchers more functional, engaging, and analytically efficient. But we agree that having accurate expert feedback is a staple, and for this reason, the feature that makes it easy to do that is scheduled for our next implementation cycle, which is the soonest it can happen.

In light of your feedback, we will look into coordinating a manual update even before the researcher panel is ready.

In a similar vein (and by way of update), our biomedical collaborators screened another 300 bad movies that were flagged by users. Those movies will be disabled tomorrow.

Thanks again, and please don’t hesitate to let us know if anything else comes up. I can’t say enough how much we truly appreciate player feedback.

Best wishes,
Pietro

3

I am too new for you to take my comment too seriously but could there be a ‘skip’ option? Sometimes I feel at a loss due to the quality of the video, placement of the frame, inability to tell which capillary is being targeted or to even see a capillary that matches the outline, etc. to have much confidence in my response. Other times I’m totally confident whether I’m right or wrong :slight_smile: you don’t need to bother to answer because I’m sure you have plenty of truly important things to do it was just a thought that would make for a little less frustration for newbies, for me at least :slight_smile:

2 Likes
4

Dear @JeanGulden,

It’s a great question. We did at one time have a “skip” button in a prototype, but our research methods require an answer, even if it is an uncertain one. We know that even when people think they don’t know the answer, on average, they have some information about the answer. So getting any response, even when you think you have no idea, actually helps the research. That’s the “quick” version of the answer :wink:

Best,
Pietro

5

Dear @gcalkins,

I’m the SC data coordinator. I just want to let you know that we just finished removing bad vessels from the new dataset. Now we are solving this problem. First we will get the opinion of few more experts on those and finally we’ll decide whether we should keep them as calibration vessels. I really appreciate your feedback, they have been great end essential help for us.

Best,
Mohammad

1 Like
6

Thanks, @mh973!

And for the ones we keep, we will also edit the expert feedback to make it more informative about why the expert answer is correct, so someone will be able to learn from that and apply that knowledge to future vessels!

Best,
@pietro

1 Like
7

You folks are amazing to be able to respond as quickly! So one more question, if I truly don’t know, no idea at all, flow or stall, is it better to guess one way than the other? We’re better to flag the video? (I’ve never done that as I thought that was just for loading problems. )

1 Like
8

Dear Jean,

Your great questions motivate us to respond - and this is another great one.

We do want you to guess when you don’t know, but we don’t want to tell you which direction to guess. That’s completely up to you :wink:

Videos should be flagged only if you think there is something wrong with them - for example, lots of frames are missing, there is “ghosting” in the image that makes it impossible to tell what’s going on, or if there is no obvious target vessel inside the outline.

1 Like
9

Hi @mh973 and @pietro
It has been almost a full month and we are all still tripping over these questionable calibrations. My count is now up to 11. I have been marking these in the comment fields. You can do a quick string search in the comment fields and find the “Questionable Calibration” items identified. I have, unfortunately, stepped on 4 of these in the last 15 minutes and my sensitivity has dropped considerably, through no fault of my own (my opinion until shown otherwise). Please address these ASAP. Comments are showing the increasing frustration by all players.

P.S. You still have many, many, many bad outlines that do not highlight specific vessels. Many players are marking these as stalled thinking it will bring these to your attention. It really means we just avoid reaching the stopping point and they remain in the data set seeking more meaningless player input.

Thanks,
@gcalkins

10

Dear @gcalkins,

Thanks again for raising this issue. I relayed the message to the whole team again just now, and I hope we can provide more feedback & deal with this soon!

Best,
Egle

11

Dear @gcalkins,

We just finished making decision about those vessels and updating the SC database. You should not see most them anymore within hours.

Regards,
@mh973

12

@mh973
That’s great news. Thank you so much.

Regards,
@gcalkins

1 Like
13

@gcalkins,

Thank you for helping us improving SC.

@mh973

14

Hi @mh973 and @pietro ,

It has been much better with several of the questionable calibrations having been removed from the dataset. Thank you.

However, we still have a few more that have not yet been addressed and I have come across 4 more that I have been able to document. (#12, #13, #14, and #15)

Let’s deal with these a few at a time.

Questionable Calibrations #6, #7, #8, and #12 are still in the dataset and are all related. Possibly we just need to improve the explanation about why these are valid calibrations for the new players so they can learn. Most of the advanced players have learned to indicate flowing for all four of these calibrations through trial and error (not that we agree). New players will normally get these wrong by indicating a stall. Here is a picture of the 4 different calibrations to which I am referring. They are all of the same vessel structure but appear to outline different segments of the vessels. It is difficult to tell when to consider the small “bridge” or “connector”. The upper and lower vessels highlighted in green are clearly flowing, but the connector is stalled.

.
This connector is such a prominent part of all of the outlined regions it will usually prompt the player to indicate a stall. Is the connector to be considered in any of these four instances? The experts have indicated that all four of these calibrations should be considered as flowing, assuming the connector is not a part of the outlined vessel segment.

I could see in calibration #8 and #12 that it is part of a branch that should be ignored. However, in calibration #6 and #7 it appears to be an integral part of the outlined center line.

Either way, these are causing problems for the newer players and my vote is to eliminate all four of them as confusing. Trying to make some of these as flowing and others as stalled would even be more confusing since they all look basically the same.

Thoughts and remarks encouraged.

@gcalkins

15

Thanks @gcalkins . I’ll get back to you soon.

16

I’m also grateful for your persistence despite this issues. I wanted to give you a bit more information about what is happening behind the scenes and why we are having difficulties with these vessels. The data for StallCatcher is organized by capillary segment, where we define each capillary as going from one branch point to another. A rather challenging part of the parceling out the data this way is making sure we get the out lines correct. This turns out to take a lot of time as we currently have someone manually trace each vessel. In many cases, we have images taken from the same mouse imaged at different times. The vessels don’t change very much from day to day, so in most cases, we can take the outline of the vessel from one imaging session and also use it for the another imaging session. Because the mouse is not exactly in the same position in each imaging session (we do our best to put it in the same place under the microscope, but it is pretty difficult), the images can be rotated or tilted. We do additional image computations to try to fix the orientation of these vessels, but as you can see, we can’t quite get it right.

In addition to trying to catch up on addressing the flagged data, we are currently working on new algorithms based on machine learning to better identify these vessel segments. However, it is likely that we will continue to encounter some of these issue. I was thinking that it might be nice for advanced players to have an option to get credit for marking problem vessels.

17

The cases above are odd cases. I think we thought that the “connecter” in red a separate vessel. And that vessel is indeed not flowing. The outlines in #6-9 are referring to the other four vessels. This is hard to see because the connecter is so short and the outlines do go over the connector.

18

Thanks @nn62 for the explanation. (cc: @pietro and @mh973 )

Let me use some annotations to make sure I have a complete understanding.

As stated earlier, all four of these calibrations are different variants of the same vessel structure shown below. The upper and lower vessels (in green) are clearly flowing (many moving blood cells) and the small “connector” (or separate vessel) (in red) does appear to be stalled.

Let’s start with #8 and #12 since these represent a common error I am seeing in the annotations. Namely, branches of the primary vessels with stalls being annotated as stalled even though the targeted vessel is flowing.

You have restated that each outline is intended to highlight only a single vessel segment (from one joint to the next joint). Even though multiple segments may be included within a given outline, it is important to identify the targeted vessel segment and only classify this single segment as flowing or stalled. In the case of Calibration #8 and #12 shown above, these targeted segments are highlighted in blue in image #8b and #12b respectively, Therefore, the observed stall (red arrow) is not within the targeted vessel segment and should be ignored. Examples of this are observed in the real vessel population. I just want to be sure that these stalled branches from the targeted vessel should be ignored.

Calibrations #6 and #7 are a little more complicated.

In these cases the outline itself inadvertently includes two different segments. (Over time you hope to improve the outlining accuracy, but for now, we will occasionally see errors like these.) Because the outline algorithm makes these occasional errors, we as players need to help interpret what was intended.

For Calibration #6 both the upper left vessel segment and the connector segment appear to be outlined as can be seen by the obvious downward turn in the outline in image #6a. I believe you are asking us to only focus on the larger vessel segment if more than one are included in the outline. The outline should have looked more like the blue outline in image #6c. In this case the smaller segment would be considered an extraneous branch (image #6d) and should be ignored.
.

I believe this is the same logic to be used for classifying Calibration #7 shown below.

The outline inadvertently included the short connector segment. The outline should have looked like the blue outline in image #7c. Again, we would be expected to ignore the stall (red arrow) and indicate this vessel as flowing.

So, in short, the “expert” explanation would be more like “Targeted vessel is flowing, extraneous stall(s) in branch vessels should be ignored.” This is true for all four calibrations identified above.

Please confirm my understanding is correct.

@gcalkins

19

Hello, Guy!

This is great stuff! I will take a stab at this…

You are technically correct that the indicated vessel segment would not be considered “stalled”. However, if someone were applying a looser interpretation and marked the connector vessel as stalled, that would not pose a serious issue for the analysis because all vessels that receive a crowd answer of “stalled” are ultimately double-checked by the scientists. Because the incidence of stalls is so low in mice with and without Alzheimer’s, double checking identified stalls does not materially slow down the research. Therefore, we can tolerate false positives more easily than false negatives.

In ambiguous cases my guidance would be to 1) flag the movie as bad, and 2) indicate “stalled” if either segment appears to have a stall.

FYI, we are working toward interface improvements that will enable vessel movies to be more easily referenced and referred back to for discussion.

Best, as always,
Pietro

20

Thanks @pietro .

There in lies the problem. You are asking us to err on the side of caution and flag all potential vessels with stalls (possibly making more false positives). However, all of the above examples are calibration movies that are marked as flowing. If we err on the side of identifying the stall, then we are penalized for doing what you are asking us to do.

If you want us to identify and mark all potentially stalled vessels, then these questionable calibration movies should be removed. We can’t have it both ways. I have already changed my behavior based on the response from @nn62 and have stopped marking stalls in branches and other non-targeted vessels. I agree that it would seem that you would rather have us mark all potential stalls, than to not mark these at all.

Either leave them in and miss more potential stalls, or take them out and ask us to mark all potential stalls we observe. Guidance is appreciated. (I have more of these questionable calibrations, but will not submit them until we have figured out the basic rules.)

Have a great week-end.
@gcalkins