Anyone who has played Stall Catchers recently might have noticed things slowing down. And thank goodness you told us, because that gave us a chance to fix it! It’s now running about 5x faster than a week ago, so if you haven’t played in a while – go log into StallCatchers.com and check out how speedy it is!
Music to my ears! Pietro
That said, we want to explain what happened. Stall Catchers is a living breathing organism – literally, because it is powered by humans. But it has a life of its own in some ways, and it is growing constantly. It’s kind of like an adolescent – it’s gaining lots of new info, but doesn’t always know the best way to use it.
Each time we add a new research dataset to Stall Catchers, so we can answer an important Alzheimer’s research question, the database gets bigger. That means when we ask the database to send you a new vessel movie, it has to search through larger and larger piles to find exactly the right one – one you haven’t seen before, one that isn’t too easy or difficult, the next most important one needed for the analysis, and so on. For anyone playing Stall Catchers, this translates into an overall slow-down in the game.
Fortunately, several of our catchers alerted us to this fact by email and comments in the online chat – some providing very useful details about the exact moment at which delays occur and how long they are.
Getting to the root cause of these issues can be challenging. Just give you an intuition about it, imagine you have a refrigerator that looks like mine – half the food has already expired . You want to maintain a balanced diet, so over the course of the day you want to have one vegetable, one fruit, one meat, and one grain. But you don’t want to get sick by eating something that might have gone bad. Also, you are always in a hurry, so you are trying to figure out the best way to pick your food quickly. What’s the best approach?
- Do you randomly choose a piece of food, and if it hasn’t expired and if you haven’t already had that food group you eat it, otherwise you put it back and randomly try again? That requires no preparation, but by the end of the day, you might find yourself picking and replacing several times before you get a non-expired version of the food group you need.
- Do you take the time to go through all the food at the beginning of each day and throw out anything that’s gone bad? That means spending extra time up front, but then at least you know anything you pick will be safe. On the other hand, you might still have to go through some trial and error to find the right kind of food for each meal.
- Do you sort each food group onto a different shelf, and then check the expiration date before you eat it? This still requires some upfront time, but then if you toss anything that has expired, you can clean up your fridge as you pick out food for meals.
- And what about avoiding waste? Do you put all the food that is about to expire on the top shelf so you waste less food by eating it before it goes bad? That adds more time to the process, but you‘d end up with more meals from the same amount of food you started with.
Each of these food selection strategies listed above ends up being affected by grocery shopping habits. What if you suddenly have five times as much fruit as vegetables? Or what if certain kinds of food have a shorter shelf life than others. What goes into the fridge can impact which food selection strategy makes the most sense.
These are the kinds of challenges we face with the movie selection process in Stall Catchers. There are lots of factors that sometimes weight against each other – user experience, analytic efficiency, and priorities on which data should be analyzed in which order to help us get an early prediction of the research results.
Occasionally when we make performance improvements to Stall Catchers, it is done situationally, in a pinch. For example, we might optimize the user experience for a big Catchathon event. In those instances, we can have near term benefits but then long term liabilities that show up as unexpected slowdowns as we near the end of a dataset. But in general, our goal is to make lasting improvements that consider the future growth of Stall Catchers in terms of participation, data volume, and research needs.
Last week, we made two changes that reduced the wait time between vessel movies by a factor of five – from about ten seconds to about two seconds. We accomplished this in two ways – first (to borrow from the analogy) we realized we were unnecessarily sorting each food group by type (e.g., fruits were being sorted into bananas, oranges, etc.) which was not necessary and took extra time. Eliminating this extra level of sorting, cut the wait time in half. The next thing we tried was equivalent to removing the expired food before sorting what was left into food groups (previously this happened the other way around). This cut the time in half yet again.
That said, this addresses only one phase of the vessel annotation cycle – the part where you click the “Next” button and wait for the next movie to load. But there are other waiting periods – such as when you click your answer and wait to see the response. I believe we can make substantial further improvements to all phases of the annotation cycle and, indeed, I am bring the Stall Catchers software code to an exclusive Microsoft Hackathon next week called “Hack for Good”, where Microsoft’s best and brightest engineers will volunteer their time for 48 hours to help us take Stall Catchers from “usable” to “blazing”!
We’ll keep you posted on our progress, but the best way to find out is to see for yourself :)
This is a companion discussion topic for the original entry at https://blog.hcinst.org/improved-speeds-in-stall-catchers/