More detailed guidance and varied examples would certainly be welcome. I’d been doing this for days before I realized what that vertical blue bar is about. But it’s no big deal. The most important thing isn’t scoring points, though that’s what makes this fun, but to improve our accuracy and save the researchers needless checking of wrong answers. The changes you mention should help with that.
P.S. 2 minor problems. The green outline around a vessel varies in brightness and sometimes I can’t see it at all. I click on Flowing and that’s almost always right. The Show answer checkbox sometimes unchecks itself, the whole screen dims, and the reload bar runs before I can do anything. I check the box again and everything is fine until the next time.
I believe the green outline’s actual brightness stays the same, but because the brightness of the surrounding area varies from movie to movie (or even within a movie), that can make it seem dimmer or brighter. We may try to make this outline more obvious in the future. Thanks for calling our attention to it!
You are absolutely right, the ratio of stalled to flowing is about 1:4. However, don’t get any clever ideas That “blue bar” that gradually goes up and down is a measure of your sensitivity. If you always click on flowing - even though your accuracy would be 80%, your sensitivity would quickly go down to zero. And since your sensitivity influences how many points are given for each annotation, it would eventually bottom-out at 1 point per vessel.
This means that, whether a person’s goal is to help the research (which I realize is your goal ) or maximize points and climb the leaderboard, or both, the best strategy in all cases is to always make the best possible decision about each vessel.
Ah yes - they mysterious unchecking check box! Apologies - this is a known issue and is on our list of things to fix. Basically, when the system senses that there is an issue loading a movie, it will automatically start loading the next movie. For some as-yet-unknown reason, when this happens, that “show feedback” checkbox gets unchecked. The temporary work-around is that if you notice your screen reloading, make sure to re-check that checkbox if you want to continue to see the feedback. This is majorly annoying, and I promise it is on our to-do list
Sorry, but no. Even when a movie “screen” is almost entirely black, the green outline can be dim or impossible to see. I then have no basis for choosing Flowing or Stalled, but I have to get on to the next movie, so I click on Flowing just to do that. Fortunately, it’s almost always right or “Maybe.” But when the expert says there’s a stall, and the red dot(s) appears, it’s in open space with no green outline to be seen. Doesn’t happen often. When it does, how can I identify the movie for you, so you can see it yourself?
That sounds peculiar indeed. The next time you come across such a case, could you please do these two things:
check the “Flag” box before submitting your answer. We go back and check flagged movies for problems (such as bad focus, vessel impossible to see, or the one you suspect here!). You could also add a comment to “Community answers” box, to indicate this was “a problem with the green outline” or something like that.
to identify the movie for us you will have to copy the page source. How you do it will depend on the browser you are using. In Chrome it is as easy as:
Right click anywhere on page → View page source
In other browsers it is generally:
Browser Menu → Developer → View source
Then select (Ctrl-A) copy (Ctrl-C) and paste (Ctrl-V) that text into an email to info@eyesonalz.com.
This will allow us to go back and check what is happening here.
Sorry for the troubles it is causing you while stall catching… We’ll see what we can do to make it better ASAP!
I don’t know if this is a problem per se, but an observation. When my blue skill bar completely filled for the first time, from that point on I only got calibration movies. I’m guessing about 15 in a row - all scored only 580 points except one that scored twice that. After I finally missed a calibration, then I started seeing non-calibration movies again, and getting higher scores. Perhaps this is by design, but just thought I should mention it. Mike C.
Since posting, I’ve seen quite a few screens in which the green outlines are so faint that they’re almost invisible, and sometimes these are covered with a large vessel. As I go through the screens, the green outline finally emerges. It may be that this is what happened earlier, that the outline wasn’t actually invisible but I didn’t look hard enough for it. But I’ll confirm that the outline does range in brightness from bold to extremely faint, regardless of what else is happening in the movie screen.
Hm, that game behavior sounds peculiar, however, are you sure those were “calibration” movies? Calibration movies = those annotated by experts (so you get a “Correct” or “Not correct” answer, and - normally loads of points); real movies = not annotated by experts, only by the community (so you get a “Maybe” and normally few points).
Therefore normally, at a high sensitivity you would get mostly “real” movies (which would score you few points), and an occasional “calibration” movie, so it sounds like this is what happened…
Did the movies you saw after sensitivity got to 100% say “correct/not correct” or “maybe”? Did you mean the 580 points were for all the movies or each ?
Yes - that can often be an issue, since there are multiple vessels crossing at other layers and blocking the view, especially if those are particularly “fat” vessels
So I guess the rule of thumb should always be: 1) scroll ahead in the movie to find the outline; 2) scroll through a bit more to find the one vessel following the shape of the outline; 3) if all else fails - flag the movie!
They were all calibration movies, Egle. Each one scored 580 points, except for the one as noted. The most I ever got for a non-calibration movie was 115 points.
Egle, I managed to claw my way back up to the top of the skill bar again. I’m pleased to report that “it” did NOT happen again. At first it seemed to be settling into a pattern (see following), but after encountering a couple ‘previously annotated’ warnings §, things randomized nicely. Here’s what I’ve seen so far with the full skill bar: 1450 cal / 3x116 real / 1450 / 3x116 / 5800 / 19x116 + 7P / 1450 / 116 / 1450 / 2x116 / 2900 / 9x116 + 2P / 4350 / 1450 / 116. I’ll let you know if I experience another strange encounter.
There was one perhaps notable difference this time vs. last. Last time I reached the top of the skill bar when I was about to level up. This time I was no where close to leveling up. Regards, Mike C.
Hi, it’s me again. Can you provide some tips for getting these right?
“Several dark spots are not moving throughout the image. Hard to see through the fluorescence.” For me, not just hard but impossible. How do you see through the fluorescence to detect dark spots?
“No black spots are seen flowing through this vessel. Therefore, it is considered to be stalled.” I see no black spots at all - the image through all the screens looks like unbroken blood flow. What are you seeing that I can’t see?
“This vessel is empty, there is no visible flow throughout.” The blood flow is definitely visible to me, or does the white stream mean something else?
Reading caprarom’s comment that he’s been at the top of the skill bar is discouraging, as I never get further than 2/3 and almost always 1/2 or lower. I assume this means that I’m probably getting 1/3 or more of the non-calibration movies wrong, and that isn’t good enough. Presently i’ve seen stalls in only 13% of the movies, which seems low. I want to do better but could use some advice.
The points system adds to the fun - thought I might catch Donna and caprarom last weekend but they’ve really taken off. Three cheers to them. But the scores can be puzzling. Sometimes I get 400 points for catching a stall, sometimes 4000; sometimes flowing earns me 300 points, sometimes 1000. Can you say what gives, or is it a trade secret?
Speaking of secrets, it would be interesting and possibly even helpful to see more than just the point scores of the other participants. What do the others think?
I agree with jfrancis—I am at Rank 18 right now because I took several days off from “catching.” I would love to see a list of all the participants—at least the top 50. I appreciate the help I get from those who post. I was blown away to find out that several have a full blue sensitive “tube.” Mine is just above the half-way mark, yet I don’t miss that many anymore. I really enjoy being a part of alz@home, and am improving all the time. Thanks, Evelyn
Hi! I can help you with your first few questions. As for the “hard to see through fluorescence” this is something that people often see after some practice. In this situation the fluorescence is very bright on either side of the stall and therefore overshadow the dark spot in a sense. If I knew what image you were specifically talking about I could be more helpful. As for the “no black spots are seen” and “this vessel is empty” actually essentially mean the same thing. The black spots are the red blood cells and white is plasma. We consider blood flow to only be the black spots (ie blood cells) so if they are not seen moving throughout the vessel, you would only see white throughout; therefore the vessel is considered to be stalled (we just cant see the immediate cause of it).
To clarify the points system, I am actually the person who annotated all the videos and as such, assigned a complexity value to them. If I considered a stall harder to find, you would get more points for being correct. For example, the “empty” vessels are ranked the most difficult to see and are therefore usually worth the most points. The more-obvious images are worth less.
Thanks much! It’s all very interesting. I guessed that the points per movie related to the blue bar - I’d get more points with a higher rate of getting it right. But now I understand it’s not me, it’s you.
Since you’ve been so obliging, one more question. Does that blue bar influence anything else, such as the Maybe points, or is it just there for show? Naturally I want to get as many analyses right as I can, regardless of points and the rest of it, but I’m curious.
You are quite perceptive. The scoring algorithm is a bit complex. The challenge is to not introduce any incentives that would bias a catcher’s response. So we don’t mean to be cagey about it, we just want to keep the research data clean.
That said, I can confirm your hypothesis that the blue “sensitivity bar” does indeed influence how many points you get for annotating any vessel.
If your sensitivity is high, you will also have an increased chance of getting a more challenging vessel to annotate. The good news is that difficult vessels, as @vpm22 pointed out, are worth a lot more. So it’s always better to do your best and raise that sensitivity bar if you are trying to get points faster
Thanks for suggesting! This has also come up in our internal discussions What did you have in mind? What else would you like to see: level, number of movies watched, sensitivity level (numerically), or something else? We don’t have much room on the Virtual Microscope (VM) screen, but we could consider adding more stats to the leaderboard page.
Have you visited the leaderboard page yet? It is one of the menu items at the top of the page. If your browser window is too narrow, then you might have to click on the “burger” icon to see the leaderboard menu option. From the leaderboard page, you can go through the full list of participants to see exactly how everyone is ranked.
Is that what you were seeking or did I misunderstand?
Also, don’t fret about the blue bar. If it is half full, you are doing better than most! It is very difficult to get it to be full, and if you do, even one small mistake can rapidly reduce the bar.
It is also worth mentioning here - for everyone - that you cannot mess up the research results. Even if your blue bar is just a thin blue line and you are only getting about half of them correct, you are still making the results more accurate. It might seem counterintuitive, but we can combine answers from 30 people who are only getting answers right a little more than half the time, and get an almost perfect answer. That’s called the "wisdom of crowd’ effect
Im not actually behind this aspect of the program, but I do believe that the bar does change your point value as well. Im not completely sure on the algorithm behind it though. From what I understand, the bar measures your sensitivity so if you are better at the calibration videos, the bar fills up resulting in higher point values.
Pietro, as I imagine it, if somebody wanted to see my stats, they’d click on my name in the list of catchers and a window would pop up with the complete stats table and sensitivity bar from my VM screen. Hit Esc or click on an x and the window would close. Their VM screen itself would remain unchanged.
Would it serve your purpose to add a third option besides Flowing and Stalled - Can’t Tell? I still can’t sometimes find the green outline at all; arbitrarily I click on Flowing to move on to the next movie, but I haven’t analyzed anything. If that doesn’t bother you, then I won’t let it bother me.
Thanks for the further inside information about the point system. I’m afraid I’m just insensitive; getting that sensitivity bar above about 2/3 is beyond me; too often, I just can’t see what the expert says is there. But so be it. Curiosity is satisfied - now back to work!
P.S. Show Answer is no longer unchecking itself. Looks like that one is fixed.
That “blue bar” that gradually goes up and down is a measure of your sensitivity. If you always click on flowing - even though your accuracy would be 80%, your sensitivity would quickly go down to zero. And since your sensitivity influences how many points are given for each annotation, it would eventually bottom-out at 1 point per vessel.
) or maximize points and climb the leaderboard, or both, the best strategy in all cases is to always make the best possible decision about each vessel.
to see the leaderboard menu option. From the leaderboard page, you can go through the full list of participants to see exactly how everyone is ranked.
Curiosity is satisfied - now back to work!