Table 2

Considering the appropriateness, organising a community steering committee, promoting the contest, assessing contributions, recognising contributors, sharing ideas and evaluating the contest: final consensus statement on crowdsourcing challenge contests in health and health research, and the consensus grade achieved by each item

ItemGrade
Stage 1. Considering crowdsourcing in health and health research
1a Before starting a project, the organisers should consider the benefits and risks of crowdsourcing in order to understand if this is an appropriate method.U
1b Crowdsourcing may be particularly useful in settings in which there are diverse networks (eg, groups, professional societies, social media movements, in-person teams) to solicit contributions.A
1c Crowdsourcing organisers should consider whether they are asking for something that would be feasible for an individual layperson to develop.U
1d Crowdsourcing organisers should ensure that they have selected an appropriate activity, based on feedback from community members and other stakeholders.A
Stage 2. Organising crowdsourcing activities
2a Before starting a project, the organisers should establish a steering committee to develop the call for entries, decide the format of submissions, and provide details.A
2b The steering committee should include people from different disciplines, including the following: (1) people who are living with disease, community leaders, civil society leaders, or other community stakeholders. (2) Key opinion leaders and network leaders who can help to distribute the contest. (3) If focused on local implementation, a member of the government or public sector. (4) If focused on research, a leader of research studies. (5) In some cases, funders as non-voting observers. (6) In some cases, private sector leaders as non-voting observersA
2c The steering committee should work together to promote the crowdsourcing activity, finalise the judging the process, develop a finalist recognition plan, finalise the prize structure, and develop a sharing plan.A
Stage 3. Promoting crowdsourcing activities
3a A crowdsourcing activity should build trust in the activity in a way that is appropriate to the local context (eg, in-person activities).A
3b A crowdsourcing activity should be promoted through social media platforms with an acknowledgement of the limitations of social media (ie, limitations on who will view and respond to social media calls).A
3c A crowdsourcing activity should be inclusive and allow contributions from diverse individuals.U
3d A crowdsourcing activity should be promoted with groups and networks of interest identified by the steering committee. Accommodation for participation of people with disability should be considered based on the purpose of the crowdsourcing activity.A
3e A crowdsourcing activity should have a clear deadline. If needed, the steering committee can extend the deadline, but this should be updated in a clear way and allow for revision for those who already submitted.U
Stage 4. Assessing crowdsourced contributions
4a The judges should provide feedback on contributions independent of each other.A
4b Criteria for selecting judges are similar to the criteria for selecting steering committee members (see above), with the additional requirement of having sufficient time to undertake judging.A
4c The contest organisers should first assess eligibility and then provide eligible contributions to judges for them to evaluate.A
4d Judges should recuse themselves from evaluating entries where there is a potential conflict of interest.A
Stage 5. Recognising crowdsourcing activities
5a Steering committee will make the final selection of finalists and respective prizes based on the prespecified criteria.A
5b Personalised announcement first: after deciding the final selection but before making a public announcement, all participants should be contacted about the decision regarding their submission.A
5c Crowdsourcing organisers should clearly explain how finalists were selected.U
Stage 6. Sharing contributions from crowdsourcing activities
6a Providing open access resources, images and templates related to the outputs from a crowdsourcing activity is important.B
6b When possible and after permission has been obtained from participants, seek permission from finalists to use their ideas and distribute them widely.A
Stage 7. Evaluating crowdsourcing through research
7a Research on crowdsourcing is important to demonstrate the value of crowdsourcing in health and health research.U
7b A crowdsourcing activity can be evaluated by using qualitative, quantitative, or mixed methods research.U