This is my second time posting this topic. I can find NO record of it being posted when I posted it before. So, will repeat. For a week now, most of the movies I’ve read have been “Maybe,” with a few points given. This morning—again—all movies I’ve read were judged, “Maybe.” The last Movie I looked at this morning had already been read by 38 Catchers who said, “Flowing.” None said “Stall.” I was #39 to judge it “Flowing.” Yet, I received a Maybe—with 60 some points awarded. I’m am VERY discouraged! Could someone explain what is going on? Evelyn
The Stall Catchers developers said that the evaluation was completed to see if the community could choose the right answer compared to the experts, and it was confirmed that crowd-based analysis is sufficiently accurate to replace the lab-based analysis. The “majority answer” by the community has not been implemented into the game yet, so right now the stall catchers community just needs to be patient until the developers can implement it.
http://blog.eyesonalz.com/whats-next/
1 Like
It’s good sign when you get to the maybes. This means that your input, along with the rest of the gamers, is being used to determined whether or not the vessel is stalled. There’s no answer provided because we don’t know yet. These are new data, for which your vote is helping determine the answer.
1 Like
Adding to what’s already been said, and I’m just another participant like you, not one of the experts. Don’t give up! The points you win are not what Stallcatchers is about, that’s just a game to entertain and motivate us. What really counts is our ratings for the movies the experts haven’t rated, for which we can therefore only get Maybes no matter how we answer.
Here’s how it looks to me - and again, I’m speaking only for myself. Now we’ve passed the evaluation phase, they’re uploading new movies which we’ve proven we can rate accurately without their help. This has to increase the odds of scoring Maybes. But as I see it, the value of my efforts here to the research is not my point score but the 15,875 movies I’ve rated so far.
All the new movies added makes it harder to score big points, and that may be discouraging especially for those who joined us later. Maybe the developers can figure out a new points system based on our ratings alone, without those increasingly scarce expert ratings. I wouldn’t think this would affect the usefulness of our work to the expert researchers. Most likely it couldn’t be applied retroactively, that would be enormously complicated, so the Stallcatchers game would have to restart from zero. That might be hard on the top scorers like Donna and caprarom, but I wouldn’t mind. What I care about is not winning an online game, which I won’t anyway, but helping to find a treatment and cure for Alzheimer’s Disease.
3 Likes
Thank you, JFrancis, I wish I could be that “lofty,” in my attitude, and I greatly admire your willingness to accept the statis quo. However, at my advanced age (88) I take my rating very personally— to me, my score is an indicator as to whether or not I am CAPABLE of being a Catcher! When I believe that my efforts are in vain, or that I am of little help in finding a cure for Alzheimer’s, I’ll step aside and let the more competent ones do it. I’ve had several replies to my Post, all encouraging, like yours is. Appreciate it. Makes me feel like a part of a Family—and that fact encourages me. Evelyn
2 Likes
Hey, Evelyn, at 75 I’m no spring chicken either. But it looks like we’re doing fine. You and I have passed the project’s evaluation stage, which means they think we are capable of being Catchers and trust our ratings.
If you want reassurance that yours are in line even when you get a Maybe, you’ve only to look down the screen and see whether your rating is consistent with the other Catchers’ - as you did! If you found 38 Catchers who said Stall when you alone said Flowing, then you might have to think about it…
1 Like
Dear Evelyn,
I really appreciate your sincere questions as well as the heartfelt answers you have already received from others catchers, all of which are basically correct.
The first thing I want to say is THANK YOU - for your tremendous contributions to Stall Catchers. And a big thank you as well to all who responded so far, including some of our most active catchers as well as Nozomi Nishimura, who is one of the key biomedical researchers on the Cornell team that is generating all of our data.
Next, I will do my best to clarify what is going on with Stall Catchers and how we intend to handle the “Maybes”. So the first thing to explain is that the better you are at catching stalls, the higher that blue bar will go, indicating your sensitivity. When your sensitivity is higher, you will get more points for ANY kind of answer.
When your sensitivity is high, that means you are providing expert-like answers so we don’t need to train or check up on you as much. In that case, you will see more vessel movies where we really don’t know the answer - in other words, the “maybes”. What it really means is that we are leaning more heavily on you for the research. But at the same time, we want to ensure you are properly rewarded for your efforts and that your score reflects this level of contribution. So when you are seeing lots of maybes, we give you additional bonuses on the calibration movies to make up for it (perhaps you have noticed this). In other words, even if you are seeing more maybes that before, your score will still go up faster overall than if you were seeing fewer maybes.
That said, because we are sampling randomly, there are time when you might see a long string of maybes, just by chance, before you see a calibration movie, which could certainly be discouraging. When you are working that hard and your answer seems to be in agreement with the consensus, to get so few points is frustrating.
We have a plan for this. We were keeping it as a surprise, but implementing this plan has taken a bit longer than we expected so I will let the cat out of the bag. We are building a new feature we call “REDEEM”. The basic idea is this: whenever you annotate one of the “maybes”, instead of giving you a small number of points for your good faith effort, we will instead hold off on giving you any points until we can turn the “maybe” into a “yes” or a “no”. Once we accumulate enough answers from other people to come up with an expert-like “crowd answer”, if your answer agrees with the crowd answer you will get as many points as if it had been a calibration movie! But if your answer doesn’t agree, you will get no points. Whenever one or more of the vessel movies you have answered is assigned a crowd answer, you will be notified, at which time you can click to redeem all your points.
Please keep in mind that when we first roll this out, it may take a while for your maybe answers to reach maturity. But once they do, you will start to see big points rolling in for all of your correct maybes. Indeed, some days you may log in and get more points by redeeming your correct “maybes” than by annotating new vessels.
So how long must you continue to tolerate the low-point “maybes”? I can’t give an exact time frame, but I can tell you it is now under active development, and we think it will be ready in a few weeks.
We thank you all for your patience and loyalty to Stall Catchers, and we have many plans in store to make this into an increasingly engaging and enjoyable pursuit. Naturally, we will continue to rely on the catcher community for candid feedback to ensure that we remain on the right track.
I hope this addresses your concerns, but if I have created more confusion than I’ve resolved, please let me know and I will do my best to explain. In the meantime, we will do our best to get the “redeem” feature rolled out as soon as possible. Frankly, I’m counting on it as a way to climb my own way back up the leaderboard. I may never catch up to all you high flyers, but my mother keeps sending me emails saying she has passed me on the leaderboard and, well, I just can’t let that go on for too long 
All the best,
Pietro
4 Likes
Oh, Pietro! What a joy reading your long-long-long email! Your Mother must be proud to have reared the helpful gentleman you have turned out to be. I had hit a down spell, stuck at #20 with no hope of ever seeing #19 again. You mentioned the Sensitivity rating I think it as my “blue tube.” The “blue tube” has never gone above three-forths full. And two days ago, after missing two in a row (with all of the rest “Maybes”) my blue tube dropped to below 1/4 full! I was ready to give up. But, after your encouraging email, I went back to work with a passion, and soon was back in the 19 slot. Blue-tube now 1/3rd full! So, perhaps there is hope for me afterall. Tell your Mom that I appreciate the way she reared you----you are kind, thoughtful, caring, and encouraging! Evelyn
1 Like
You are too kind, Evelyn 
But if I tell my mother, I will never hear the end of it! It’s bad enough that she’s beating me at my own game. 
I do realize the “blue tube” (I love your term for it!) can drop suddenly and then take a while to go back up, so I am glad you are steadily rebuilding yours! Thanks for giving it another chance.
Stay tuned for the “redeem” feature - coming soon!
Best, as always,
Pietro
[quote=“pietro, post:9, topic:113”]
“Stay tuned for the “redeem” feature - coming soon!”
OH! Pietro! I just received my “redeem” points–and I am now a “millionaire!” With 1187429 points! That puts me at Rank 17, and I am no longer “discouraged by the ‘Maybes.’” You guys are the greatest. You not only work yourselves to death compiling the results of our labors in judging movies, you go the extra mile, in giving credit where credit is due! THANKS! Evelyn
2 Likes
I just saw the big blue Redeem button, clicked it, and wow! What a bonus! I won’t boast about how many points, but it moved me up a place in the standings. Probably just temporary, though, until Tom Adams redeems his Maybes.
1 Like
Hey, guys, some bad news from here—I fell and apparently ruptured L-3 disk in my back. Will see what MRI shows on Monday morning (Feb. 27). In meantime, can’t spend much, if any, time on Alz@home. Just sick about it, but my doc has grounded me—little or no computer work until later. Pray for me, I love Alz&home! Evelyn
2 Likes
We are so sorry to hear about your recent fall and injury, Evelyn, and wish you a comfortable and speedy recovery. Good luck with the MRI as well. You are surely in our thoughts and prayers.
Come back to us soon! (you don’t want to fall behind too much on the leaderboard , plus we have something new in the works…)
My best,
Pietro
1 Like
Discouraged a bit here. Pretty new at this. So, how can I tell for sure if I am doing well enough for it to be helpful? Sometimes I do disagree, even with the experts, and am at a loss to find a way to see their point of view or learn from my error. ( Ex. they say stall and I see white in the next frame or previous frame, right where their red dot was?) How high is ‘good’ on my blue tube and how high is great? Where can I go to learn even more than the training videos I found? Is there a ‘help me with this one’ option?
I am very excited to find a possibly useful way to spend my “just want to be challenge a little in play” time. Still, I need to feel what I am doing is actually correct to continue 
1 Like
Dear @JeanGulden,
Thanks for the beautifully articulated questions! Please see below…
[quote=“JeanGulden, post:14, topic:113, full:true”]
Discouraged a bit here. Pretty new at this. So, how can I tell for sure if I am doing well enough for it to be helpful?[/quote]
Sorry about the discouraged feelings! I think this is common and so we are working on ways to make it easier for folks to play the game and see the contributions they are making to the research.
We believe a few expert answers are actually incorrect (it looks like you already found the relevant thread: http://forumhcinst.org/t/dealing-with-questionable-calibration-movies/158/4). But some are correct and just hard to understand. We have an “advanced” tutorial under development for folks who understand the basics of annotation and want to go to the next level. This was created by our ground truth expert, and is quite revealing. It requires some editing, but we will aim to post it in the next month (note to: @Claire).
If any blue is showing at all, you are making a research contribution. If the tube is half full, you are helping a lot! Because of our analytic methods, if you are right even 51% of the time, you are helping the research.
Please see above. The “expert” tutorial will be coming out soon! Part of what’s involved is understanding what you see not just visually, but also conceptually. The new tutorial will help with that. One key concept is that each vessel in an outline joins two other vessels. The flow dynamics of the joined vessels sometimes helps us understand what is happening in the target vessel.
I completely understand. The bottom line is this: 1) you can’t hurt the research by guessing, 2) everyone who plays makes a beneficial contribution to the research, 3) over time, you will learn to make each of your annotations even more valuable (as measured by the blue bar). But as I said, if there is any blue at all in your bar, your answers are helping as get closer to a treatment.
Thanks again and best wishes,
@pietro
2 Likes
