Hello everyone!
Today, we have exciting news about our continuing work on human computation, citizen science, and ethics!
In the article “Human Computation Requires and Enables a New Approach to Ethics” which has been presented at the NeurIPS 2020 Crowd Science Workshop: “Remoteness, Fairness, and Mechanisms as Challenges of Data Supply” last week, we – Patricia Seymour, Pietro Michelucci and Libuše Vepřek – discuss the need for establishing a set of ethics to ensure the fair
treatment of online cognitive laborers and citizen scientists as well as the conscientious use
of the capabilities to which they contribute. Moreover, we introduce a dynamic approach for a sustainable ethical framework.
This work is also based on the great and fruitful discussions we’ve had in this forum over the last months and in the workshop in September and we are very grateful for your contributions! We don’t want to stop here but hope that we can contribute to the further discussion and would like to encourage you to participate.
We look forward to hearing your thoughts!
I read the article, and I’m confused about some things. First off, sometimes it’s not clear to me if the volunteers are researchers, or participants in the research. Most of the time it seems to me they are volunteer researchers, but in the study where we were evaluating the behavior of the bot, it seems that we thought we were testing the bot, but the bot was actually testing the volunteers performance. If data is actually being collected about the performance of the volunteers, in other words if they are actually the subjects of the study, a different type of ethical review would be necessary. For example, I don’t recall that there was any debriefing after we tested the performance of the bot in that study. I thought the bot was making a lot of errors, but that was actually intentional in order to test my performance.
At one point you say, ethical review is something that we demand for our legal system. I’m not sure which legal system you are referring to, I suppose it doesn’t matter, but it did throw me a bit.
When you talk about using a sandbox. I’m not really sure what the sandbox is referring to, but again I’m not sure if it’s important or not.
What I found most confusing was your flowchart. I’m not sure which direction things are supposed to be going. I see all the arrows, but I don’t understand what happens first, second, or third. Do things proceed in some kind of sequential order?
Does everything happen at the same time? I see that the IRB expert is thinking about a flowchart, but I don’t understand how that relates to the rest of it. If each of these steps relates to a particular order in the review process, then some numbers on the arrows would be helpful. If all the processes occur at the same time, then that’s fine, but I think that idea needs to be more clearly stated in the discussion. It’s not really clear to me what’s going on in the flowchart.
I hope you are well and thank you for your thoughtful comments / questions. I asked @LVep to join me in answering this, so this is a collaborative response
The question about whether volunteers are researchers or participants in the research is a very good one and we tried to show in our article that the answer is not straight forward when it comes to online citizen science and human computation precisely because volunteers can take different and sometimes also multiple roles. So in each new context, roles have to be explained and options given. We argue that this is currently not considered enough and that systems and also review processes have to be more flexible to take into account that the roles can vary. This means that we completely agree with your argument that different types of ethical review are necessary in the different contexts.
With the “sandbox” we refer to an environment where a clone of a project such as Stall Catchers can be run to conduct experiments without affecting the project itself. For example, the study you participated in and mentioned in your comment was conducted in the “sandbox”, which is why it had a different look and feel.
Thank you for your question about the flow chart and the order of the different steps. It is indeed difficult to define a strict sequence of steps due to various interdependencies (for example, a certain response at one step might cause us to return to a previous step). But the idea is that if a new project has to be reviewed by an IRB expert, then the IRB expert can access the sandbox to test the project itself and understand the individual steps and processes of the project (that’s the flowchart the expert is thinking about in our flow chart). If questions come up the IRB expert can easily contact the researcher and ask questions or discuss aspects with them (it’s a dialogue - speech bubbles in the flow chart). One important new aspect of the proposed IRB process is that the community, which consists of citizen science participants, researchers, project designers and other involved parties, defines what values are important to them and these values are then included into the ethical guidelines (arrow from community to ethical guidelines in flow chart). This means that the ethical guidelines consider the project- and context specific needs of all involved parties – especially the needs of the volunteers – and are dynamic, that is, they are adapted over time with changing values and norms. So if an unforeseen situation arises that doesn’t fit our current ethics, the community weighs in on what to do about it. The IRB expert then evaluates the project based on these ethical guidelines, informed by the community (arrow from ethical guidelines to IRB expert).
If an aspect comes up in a project to be reviewed by the IRB expert, which is not addressed in the current set of ethical guidelines, then the IRB expert can contact the community to review the issues properly (arrow from community to IRB expert in flow chart).
Moreover the system logs the (ethical) issues that arise and the ensuing dialog around those issues, as well as any implemented solutions. The protocol repository not only allows researchers to use designs and approaches that have already been approved (save time, minimize redundancy in review processes) but it can also be accessed by e.g., the community (arrow from repository to community).
Although we cannot provide a precise order of the individual steps, we hope that this explanation clarifies the procedures.
Thanks again for your engagement and for these great questions!
That’s all very interesting. It’s not quite clear to me how the system would log the ethical issues that arise. Are you talking about an actual computer program, or is that just a written protocol that you would consult in such a situation?
Also, is not quite clear to me what the protocol repository is. A concrete example of that would help. What comes to my mind is that you were able to devise stall catchers based upon the Stardust platform, but in that situation, you were copying more of the data collection apparatus rather than dealing with the ethical issues, but I suppose that the Stardust data collection would have faced similar ethical issues if they ever even bothered to address them. I guess a better example is that you’re able to build the dream catchers platform based upon your work with stall catchers, but even there I’m not even sure if the ethical issues for one project would have mapped perfectly well onto the other.
I guess what I’m saying is that concrete examples always help, and it’s difficult for me to visualize what situations you are referring to when we are just talking about things in the abstract. However, I suppose until these situations arise it will be difficult to have a concrete discussion about what just exactly they may be. In any event, thank you very much for your reply to my questions.
Thank you very much for your reply!
Those are all very good questions and I would love to provide more concrete examples. However, currently, most of the ideas actually are still abstract and haven’t been implemented yet. We are now starting to build this system and would of course be happy to give some more concrete examples etc. along the way. We will certainly keep you up to date!
Best,
Libi
