How does decision-making relate to stall catching?

I have some questions about how decision-making effects the stall catchers game. I read an article in the July\August issue of Discover magazine about decision-making.

There are two different systems used in decision-making. System one is referred to as the fast system. The basal ganglia, and cortex play a major role in it. Groups of neurons develop patterns of firing that are strengthened with repeated exposure to stimuli. This is key to making split-second decisions under pressure.

System number two is referred to as being slow. The prefrontal cortex, and the hippocampus are involved in the system. They work together in situations that involve rule-based decision-making.

One of the drawbacks of the fast system is that it leads to more errors than using the slow system. In a stall catcher competition, like the one that’s going to be held next week, would it be more likely that players would make more mistakes because a time limit is involved in the competition situation? In other words, would players be more apt to use the fast decision-making system as opposed to the slow decision-making system during a competition?

Experts make decisions differently from novices. In one study that I read about, fire chiefs were interviewed to see how they make decisions. It was thought that the fire chiefs would provide a limited range of options, and then carefully weigh the pros and cons of each option. However, it was found that the fire chiefs just knew what the correct approach would be without having to weigh the options. They could do this because they had years of experience about how to fight fires, and were able to do template matching in which they were able to match the elements of the current situation to elements of situations that they had experienced in the past.

I also read somewhere that chess masters can do a similar type of template matching when they evaluate a chess position. They have hundreds, or even thousands of chess positions stored in their brains. When they look at a chess position on the board, they can match it up with previous games that they have either experienced, or studied. On the other hand, chess novices, usually have to evaluate every possible option on the board before making a move.

How do stall catcher experts like the ones who provide us with the correct answers approach a particular blood vessel? Do they need to carefully look at each frame of the movie in order to make a decision the way the players do, or can they do a kind of template matching based on previous experience like the experts in these other fields?

How accurate are the judgments of the stall catcher experts? How much more accurate are the stall catcher experts than the players? In the stall catcher game we use the concept of sensitivity. What is sensitivity exactly? Are sensitivity and accuracy essentially the same thing, or are they different concepts altogether?

How do stall catcher experts like the ones who provide us with the correct answers approach a particular blood vessel? Do they need to carefully look at each frame of the movie in order to make a decision the way the players do, or can they do a kind of template matching based on previous experience like the experts in these other fields?

A template-matching kind of decision making is a good way to describe the expert annotations, these individuals have been looking at stalls and movies as well as taking an active role in imaging for years.

How accurate are the judgments of the stall catcher experts? How much more accurate are the stall catcher experts than the players?

In many cases the expert can be confident in an answer (that others may be less confident in) because of some background knowledge about image acquisition. We have been discussing the possibility of presenting additional explanatory information as players reach higher levels, as well as parsing the movies into various difficulty levels (taking into account nuanced reasoning behind why something is stalled when it isn’t obvious). While we are still in the discussion phase of this development, what it could look like would be “unlocking” harder movies along with more information on the mechanisms behind imaging, explanations of dyes, etc. as you climb levels. We didn’t include this originally because many players are not interested in needing to learn a bunch of background to understand how best to play StallCatchers, and we don’t want to scare people off with the thought that the game will take forever to learn and get good at. You are the perfect example of someone with the opposite mindset however, and we could benefit from getting your input when this concept is a bit closer to implementation stage.

In the stall catcher game we use the concept of sensitivity. What is sensitivity exactly? Are sensitivity and accuracy essentially the same thing, or are they different concepts altogether? would it be more likely that players would make more mistakes because a time limit is involved in the competition situation? In other words, would players be more apt to use the fast decision-making system as opposed to the slow decision-making system during a competition?

@pietro

Sensitivity in the game is actually d-prime from signal detection theory. It’s actually not quite the same as accuracy (though under certain conditions, it can reflect accuracy).

Consider a game where you go through a deck of cards and have to guess whether each card is a “number card” or a “face card”. Because there are more (40) number cards than face cards (12), if you always guessed number cards you would be more accurate (40/52 = 77% accurate), where if you always guessed face cards, you would only be 23% accurate. However, with our “sensitivity” metric, whether you guessed “number card” all the time or “face card” all the time, your sensitivity would be 0 either way because it is just guessing.

You can imagine how this would be important when stalls in real vessels tend to be so rare. If we just used accuracy, some people (even well-intentioned people) might guess “flowing” all the time to rack up points and we wouldn’t learn very much about the data.

Best,
Pietro

Could you explain to me what the point is of the calibration blood vessels? We get a lot of points for getting a calibration blood vessel correctly annotated. What’s really being measured is our sensitivity to be able to recognize the target, in this case a stall in a blood vessel, but it seems like most of our points come from correctly identifying a calibration movie, not a real one.

If the whole point is to correctly identify the real blood vessels that are stalled, why do we get so many points for correctly identifying the calibration movies? I think I understand the card deck analogy, but where do the calibration movies come into play, and why are they repeated so many times?

Is the point that there is a concern that our sensitivity could decrease after seeing a certain number of real movies, so our sensitivity needs to be continually measured to ensure that it has not decreased? If that is the case, then why are there sometimes two or three calibration movies in a row?