You are here

Article Text

Article menu
Epidemiology
Original research
Inclusion of participants from low-income and middle-income countries in core outcome sets development: a systematic review
  1. Jamlick Karumbi1,2,
  2. Sarah L Gorst1,
  3. David Gathara2,3,
  4. Elizabeth Gargon1,
  5. Bridget Young1,
  6. Paula R Williamson1
  1. 1Health Data Science, University of Liverpool, Liverpool, UK
  2. 2Health Systems Research, KEMRI-Wellcome Trust Research Programme Nairobi, Nairobi, Kenya
  3. 3Centre for Maternal, Adolescent, Reproductive & Child Health (MARCH), London School of Hygiene & Tropical Medicine, London, UK
  1. Correspondence to Dr Jamlick Karumbi; J.Karumbi{at}liverpool.ac.uk

Abstract

Objective Our study aims to describe differences or similarities in the scope, participant characteristics and methods used in core outcome sets (COS) development when only participants from high-income countries (HICs) were involved compared with when participants from low-income and middle-income countries (LMICs) were also involved.

Design Systematic review.

Data sources Annual Core Outcome Measures in Effectiveness Trials systematic reviews of COS which are updated based on SCOPUS and MEDLINE, searches. The latest systematic review included studies published up to the end of 2019.

Eligibility criteria for selecting studies We included studies reporting development of a COS for use in research regardless of age, health condition or setting. Studies reporting the development of a COS for patient-reported outcomes or adverse events or complications were also included.

Data extraction and synthesis Data were extracted in relation to scope of the COS study, participant categories and the methods used in outcome selection.

Results Studies describing 370 COS were identified in the database. Of these, 75 (20%) included participants from LMICs. Only four COS were initiated from an LMIC setting. More than half of COS with LMIC participants were developed in the last 5 years. Cancer and rheumatology were the dominant disease domains. Overall, over 259 (70%) of COS explicitly reported including clinical experts; this was higher where LMIC participants were also included 340 (92%). Most LMIC participants were from China, Brazil and South Africa. Mixed methods for consensus building were used across the two settings.

Conclusion Progress has been made in including LMIC participants in the development of COS, however, there is a need to explore how to enable initiation of COS development from a range of LMIC settings, how to ensure prioritisation of COS that better reflects the burden of disease in these contexts and how to improve public participation from LMICs.

  • epidemiology
  • public health
  • protocols & guidelines

Data availability statement

All data relevant to the study are included in the article or uploaded as online supplemental information.

https://creativecommons.org/licenses/by/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/.

https://doi.org/10.1136/bmjopen-2021-049981

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • This is the first paper describing the differences and similarities in the core outcome sets (COS) development processes when there are participants from low-income and middle-income countries (LMICs) compared with when participants are exclusively from high-income country settings.

    • This paper describes COS published up to 2019. It is possible that more COS involving LMICs participants have been developed during 2020 or are in the process of being developed now.

    • Public participation has been described in only a few studies, this may introduce a possible bias in the classification of public participants categories thereby altering the conclusions of public participation.

    Background

    A key barrier to translating clinical research into practice is lack of conclusive evidence. This is due to different trials, of the similar interventions, being too heterogenous and reporting on outcomes that are often non comparable1 2 or not relevant to patients and users.3 4

    Development of core outcome sets (COSs) could ensure all future research in a field reports a common subset of outcomes. This would reduce research waste thereby enhancing comparability and improving research translation and use.5 6 COSs are agreed-on minimum standardised outcome sets that should be measured and reported in all research in a given health area.7 They consist of a core domain set (this defines what domains should be measured) and core outcome measurement set (defines the instruments which would be appropriate to measure the domain).8 In recent years, the use of COS has been promoted by journals, Cochrane review groups and funders.9 10 COSs have also become useful in routine clinical practice data collection for clinical audit and feedback.11 12

    Since 2010, the Core Outcome Measures in Effectiveness Trials (COMET) Initiative has been collating, stimulating and promoting the development and use of COS by maintaining an up to date publicly available searchable database.13 The COMET database includes published studies of COS development, as well as planned and ongoing work. The types of studies included in the database are those in which COS have been developed, as well as studies relevant to COS development, including systematic reviews of outcomes. Within the database, there are publications that describe a COS only (hereafter denoted specific for COS), and other publications that describe a COS as well as provides recommendations about other trial design aspects like eligibility criteria (hereafter denoted as COS as part of a wider trial design). Studies describing the development of a patient-reported outcome (PRO) COS (a core set of patients reported symptoms and health-related quality of life domains) or core event set (a core set of adverse events or complications). Initially, the database included studies that had been identified by ad hoc means, however, it is now systematically updated using the annually updated systematic reviews as the basis of the updates since 2013.14–17 The use of this database should help minimise duplication of effort in the development of COS and potentially encourage the use of COS to tackle global health challenges.

    How are COS developed?

    There is no gold standard for development of COS, however minimum standards have been agreed in the last few years. These are the Core Outcome Set-STAndards for Development (COS-STAD) recommendations, comprising 11 minimum standards for all COS development projects.18 The recommendations focus on three key domains: (1) the scope of the COS; specific area of health or healthcare that the COS applies needs to be described, with details of health condition, population and types of interventions. The COS may be developed to encompass all stages or severity of a health condition or it may be focused on a particular sub-population, for example, a COS may be developed for all patients with COVID-19, or it may focus on patients with long COVID-19 only. (2) the stakeholders involved in the COS development; the stakeholder groups to be involved and the target number from each group is dependent on the scope of the COS and the existing knowledge and practicability. Decisions on stakeholders’ involvement should be documented and explained in the study protocol. (3) the consensus processes; this typically involves developing a consensus on ‘what’, ‘how’ and ‘when’ to measure. Some of the methods used to build consensus include: Delphi technique; nominal group technique, consensus development conference and semistructured group discussion. The Delphi technique with two or three rounds combined with a consensus conference has been most widely used.19–21 Choice of method is informed by several factors including; the need to build a true consensus with methodological rigour, strategies to ensure that a diverse range of opinions are heard, and factors such as financial and carbon costs that might limit the practicality of face-to-face meetings.7

    To date, COSs have been developed for various thematic or disease conditions and more are being developed.13 19 It has been opined that development and use of a COS would help reduce outcome heterogeneity and reporting bias, while ensuring that wide-ranging perspectives, including patients’ opinions, are incorporated, thus enhancing the value and quality of research.22 Most COSs have been developed from the perspectives of high-income countries (HICs), with over 70% of COS including participants from Europe and North America. There has been an increase in the proportion of COS that have included participants from Africa and South American countries in COS development, but this still remains low at around 25% as per COS published up to the end of 2018.19 A closer look shows that the proportion of COS participants residing in low-income and middle-income countries (LMICs) of these regions is even lower being just about 16% in 2016. Further, participants have mainly been those with research and clinical expertise in the various disease areas rather than as public participants with relevant lived experience.16

    Previous work describing COS studies14–16 19 has not examined whether there is a difference in the scope, stakeholder participation and approaches to consensus between COS that have include had participants only from HICs compared with those that also included participants from LMICs. The differences in the global burden of diseases may influence which COS are developed, and available resources may determine the methods used in COS development. For example, LMICs have had a higher level of infectious diseases while non communicable diseases have been generally more prevalent in HICs. The hypothesis therefore is that COS with LMIC stakeholders are more likely to be developed for infectious disease conditions and those with predominately HIC stakeholders to are likely to be for non-communicable diseases. Additionally, different methodologies for consensus building require different level of resources, for example, a face-to-face Delphi workshop may be more expensive if it must include a wider range of stakeholders and online Delphi processes are also likely to be influenced by internet connectivity which may be lower in LMICs compared with HICs. The aim of this paper is to describe the differences or similarities in the scope, participant characteristics and methods used in COS developed with participants from HICs and LMICs as classified by the Organisation for Economic Co-operation and Development.23

    Methods

    Study selection

    Inclusion criteria

    We included all COS studies, identified from the original COMET systematic review and annual updates.14–17 24 25 Studies were eligible for inclusion in the review if they developed or applied methodology for determining which outcome domains or outcomes should be measured in research clinical trials or other forms of health. Studies were eligible for inclusion if they reported the development of a COS, regardless of any restrictions by age, health condition or setting. Studies describing the development of a PRO COS (a core set of patients reported symptoms and health-related quality of life domains) or core event set (a core set of adverse events or complications) were also eligible for inclusion. Studies describing the update of an existing COS are included as linked papers to the original COS.17 Eligible studies are added to the database, as they are found with the annual update to the systematic review, ensuring the database is kept up to date.

    Identification of relevant studies

    The search strategy for identifying eligible COS development studies has previously been described14–17 and is briefly described here. A comprehensive search is used to identify studies that had been published or indexed in SCOPUS and MEDLINE via Ovid from inception up to December 2019. The search strategy, which was developed for the original COMET systematic review in 2013, is provided in online supplemental appendix. Annual database searches were repeated in 2015–2020 to identify studies published up to the end of 2019. Hand searching of studies is also performed.

    The review was conducted in accordance with the Preferred Reporting Items of Systematic Reviews and Meta-Analyses (PRISMA) guidelines and the included studies are shown in a PRISMA flow diagram (see figure 1).

    Figure 1
    Figure 1

    Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow chart of identification of eligible studies from the COMET database. Data were extracted from the COS systematic reviews.14–17 COS, core outcome sets; COMET, Core Outcome Measures in Effectiveness Trial; HIC, high-income country; LMICs, low-income and middle-income countries.

    Data extraction

    Data were extracted in relation to scope of the COS study (aim of the study, intended use of the COS, year of publication, target population, intervention and disease or condition area), participant categories (clinicians, researchers etc, and their geographic location) and the methods used in outcome selection (Delphi, interviews, focus group discussions, etc, alone or in combination). Data on the burden of disease in LMICs for the last 5 years were obtained, from the Institute of Health Metrics and Evaluation database,26 to help check its alignment to the COS developed over the same period.

    Analysis

    A description of the proportion of COS that included participants from LMICs and those which were initiated from the LMIC setting is provided. For a COS to be classified as being initiated from an LMIC, either ethical approval of the study had to have been sought in an LMIC, and/or the affiliation of the lead author had to be based in an LMIC; we also used the sites of participant recruitment as confirmation. The scope, participant categories and consensus methods used are described for COS that included participants from LMICs and those with exclusively HIC participants separately. Participation here is defined as being a study participant and contributing data to the COS study.

    Patients and public involvement

    Patients and the public were not involved in the development of the research question or the design and conduct of this systematic review.

    Findings

    General description

    There was a total of 550 references published by end of 2019 in the COMET Initiative database describing 370 COS. Of these 370 COS, 295 (80%) included only HIC participants and 75 (20%) also included LMIC participants. Two COS had only LMIC participants, one in Sri Lanka27 and another in China.28 A flow diagram of the study selection process is provided in figure 1 as guided by PRISMA statement.29

    Year of publication

    Between 2014 and 2019, there has been an almost twofold increase in COS that have included LMIC participants to 28% (40/141) from 15% (35/229) in preceding years. Figure 2 describes the absolute numbers of the COS developed up to 2019.

    Figure 2
    Figure 2

    Year of COS publication. COS, core outcome sets; HIC, high-income country; LMIC, low-income and middle-income country.

    Scope of included studies

    The comparison of the scope of the included studies is shown in table 1. Overall, more than half (63%) of the studies were specifically designed for COS development rather than being part of a wider project or trial. However, a higher proportion of studies with LMIC participants were specific for COS (83%) compared with those with only HIC participants (58%). Over 80% of COS were developed for research only. More than half of studies (55%) did not explicitly report on the age group of the target population characteristics. During the interpretation of the data, this was assumed to be adults. This was based on the disease being predominantly associated with the adult population and the lack of reference to any literature on children.

    View this table:
    Table 1

    Scope of included studies

    Cancer (16% for COS with HIC participants only and 12% for COS with LMIC participants) and rheumatology (9% for COS with HIC participants only and 16% for COS with LMIC participants) were the dominant disease domains across the COS developed either with LMIC or HIC participants. A slightly higher proportion of COS for rheumatology, pregnancy, skin, anaesthesia and child health included participants from LMICs compared with HICs. Five COS on health systems had LMIC participants. Figure 3 describes the disease categories that were covered by the COS.

    Figure 3
    Figure 3

    Comparison of disease categories covered in the COS with exclusively HIC participants and those also with LMIC participants. COS, core outcome sets; HIC, high-income country; LMIC, low-income and middle-income country.

    Of the 75 COS with LMIC participants, only four (5%) were initiated from LMIC setting. All were from Asia, with two from China28 30 and one each from Sri Lanka27 and the Philippines.31 No COS has been initiated from Africa or South America, although COS for conditions which are more prevalent in LMIC settings had a higher number of participants from LMICs. For example, the malaria COS by Moorthy et al32 included participants from ten sub-Sahara African Countries but none from other LMICs. The COS for the prevention of preterm births had LMIC participants from Egypt, Nigeria, South Africa, China, Iran, Lebanon, Pakistan, Argentina and Brazil.33

    Characteristics of participants involved in consensus process

    All COS that had LMIC participants explicitly reported characteristics of their sample compared with 92% of COS that had only HIC participants (see table 2). Overall, over 70% of COS explicitly reported including clinical experts; this was higher where LMIC participants were involved (92%). A slightly higher proportion of COS that included LMIC participants (57%) had included members of the public compared with those with HIC participants only (32%). A higher proportion of LMIC COS (23%) had carers as part of public participants compared with the HIC COS (9%). Of the 28 COS with LMIC participants, only 4 (all with HIC patient participants too) translated the COS development materials to non-English languages. These are COS for clinical trials in acute diarrhoea,34 infant colic,35 childhood constipation36 and type 2 diabetes.37 Non-clinical research expertise was more common in COS with LMIC participants (44%) compared with those with HIC participants only (31%).

    View this table:
    Table 2

    Participant characteristics

    Participants from LMICs were drawn from the three continents of Asia (60%), South America (49%) and Africa (31%). However, there was a skewed representation with LMIC participants largely being from three countries, with South Africa representing 50% of participants from Africa, Brazil representing 46% of participants from South America and China representing 30% of the participants from Asia.

    Methods for selection of outcomes

    Methods used for outcome selection ranged from Delphi, consensus conferences and semistructured discussion as shown in table 3. Rarely was one method used for COS development. Semistructured discussion seemed to be the most commonly used single method, with 20% of COS with HIC participants only and 8% of COS with LMIC participants describing its usage.

    View this table:
    Table 3

    Methods used in the selection of outcomes

    Most COS used a combination of methods. Literature review in combination with other methods was most common, with 37% of COS with HIC participants and 60% of COS with LMIC participants using this method as shown in table 3. Only one COS that included LMIC participants had no methods described compared with six that only had HIC participants.

    A combination of the three most common methods was more likely to be used for LMIC COS (n=23, 30%) compared with the HIC COS (n=37, 13%) as shown in figure 4.

    Figure 4
    Figure 4

    Combinations of methods for outcome selection. HIC, high-income country; LMIC, low-income and middle-income countries.

    Discussion

    This review provides the first comparison of COS that have been developed with participants from HICs and LMICs.

    Out of the 370 published COS, 75 (20%) COS included participants from LMICs. Thirty-three COS have been published since the fifth update of COS systematic review with fifteen including participants from LMICs.19 The COS span a range of health areas, target populations and methods used. The methods for outcome selection are similar across the settings, perhaps reflecting the increased standardisation of methods of COS development generally.

    Interestingly, even though participants from LMICs have been involved in earlier COS, for example, the cancer reporting recommendations by Miller and others in 1981,38 the inclusion of LMIC participants has increased predominantly in the last 5 years. In this time, there has been an almost twofold increase in the proportion of COS that have had LMIC participants. Global connectivity is allowing people to conduct research without the need for physical meetings. A look at the development processes used is ‘older’ COS shows that most depended on consensus building during conferences and physical workshops.32 38 39 These kinds of consensus meetings tend to be more expensive and limit inclusion of participants from varied settings. Despite the increase, Brazil, China and South Africa have been the main drivers of participation from LMICs.

    All the COS developed across the settings have mainly focused on research or research and practice. A slightly higher proportion (19%) of COS with LMIC participants are for research and practice compared with 11% of COS with HIC participants only. Given that COS with participants from LMICs are ‘newer’ this could reflect that more recent COS are being developed with the intention of spanning the continuum from research to routine practice in mind and also recent availability of guidance for COS development like the COMET handbook11 and COS-STAD18 might have influenced the inclusion of wider groups of stakeholders.

    Although the COS span a wide range of conditions, the top five conditions where COS exist are cancer, rheumatology, neurology, cardiovascular and orthopaedic conditions. These are mainly non communicable diseases, and even though these are now becoming more common in LMICs, the burden of these conditions has mainly been in HICs.26 Of the COS with LMIC participants, pregnancy, lungs and health systems were among the top five conditions. LMICs are still facing infectious diseases like neonatal sepsis, lower respiratory tract infections, malaria and HIV among others,26 40 and one would therefore expect that COS with LMIC participation would reflect this. Only two COS, for malaria32 and leprosy31 represent conditions that are usually of high burden in LMICs. Thus, even though LMIC participants are being included in COS development, COS for research may not be reflecting the disease burden in LMICs. This may be because research funding priorities, for which COS are then being developed, are more likely to reflect an HIC perspective.

    Interestingly, only four COS have been initiated from an LMIC setting, two from China (COS for hip-preserving treatment of osteonecrosis of the femoral head,28 COS for infertility treatment research30) and one each in Sri Lanka (COS for effectiveness of antiepileptic therapy in children27) and the Philippines, (COS for tibialis posterior transfer surgery31). There is still no COS initiated from Africa or South America. Given the varying disease epidemiology and burden between HICs and LMICs there is a need to identify mechanisms to enable LMIC stakeholders to develop context relevant COS so as to improve applicability and adoptability of COS to these settings.14 These strategies could probably include awareness raising about COS in LMIC settings, provision of resources for COS development and dissemination of guidelines and methods for COS development. These are some of the barriers which have been described potentially affecting COS development from an HIC perspective (interviewees were mainly from Europe and North America).41 There is a need to assess which barriers and enablers are particularly relevant to COS development in LMIC settings.

    Details of public participation are still suboptimally reported, and where reported, a higher proportion of COS with LMIC participants had some public participation compared with COS with HIC participants only. Of note however, is that public participants tended to be from HIC settings. For example, the type 2 diabetes COS included public participants from the UK and Greece,37 the breast cancer COS included patients with breast cancer from USA42 and the COS for acute diarrhoea,34 infant colic,35 childhood constipation36 all included public participants from Europe. Public participation from LMICs could potentially be improved by translation of consensus building materials into non-English languages. Harman et al translated Delphi materials into Portuguese language in a two times a day to improve public participation from Brazil.37 Unfortunately, there still were no public participants from Brazil suggesting that there are additional reasons for the lack of public participants from LMIC settings.

    The lack of public participation from LMICs could be due to numerous reasons some of which could include temporal, cultural and resources differences between LMICs and HICs. First, COS that have included participants from LMICs have mostly been developed in the last decade, during which guidance has been published on how to develop COS including patient participation. Second, the availability of organised patient groups in HIC settings for given conditions or diseases provides a means of identifying and inviting patients as public participants; such groups are generally lacking in LMIC settings. Third, information asymmetry between the public and patients versus clinicians or researchers could potentially limit the participation of the former in COS development. Although evidence is scarce, we suggest that information asymmetry may be a bigger problem in LMICs due to lower literacy levels when compared with HICs.43 Additionally, some of the major research funders in HICs, for example, the National Institute for Health Research in the UK and the Patient-Centred Outcomes Research Institute in the USA, are increasingly championing public participation in research.

    It is therefore likely that a multipronged approach is needed. Further research to explore what would work, in addition to language translation, to improve public participation from LMICs is also needed.

    Additionally, there is need to describe the stage at which public participants are involved in COS development and whether it influences which outcomes are included in the COS in a given area. Previous work has shown that patients might sometimes be inhibited when put together with clinicians in a consensus building process44 but recent experiences show this is changing, with high satisfaction reported, at least from HIC countries, where joint consensus meetings are held.37 45

    This is the first paper describing the differences and similarities in the COS development processes when there are participants from LMICs compared with when participants are exclusively from HIC settings. However, there are a few limitations. First, this paper describes COS published up to 2019. It is possible that more COS involving LMICs participants have been developed during 2020 or are in the process of being developed now. Second, public participation has been described in only a few studies, this may introduce a possible bias in the classification of public participants categories thereby altering the conclusions of public participation.

    Conclusion

    Progress has been made in including LMIC participants in the development of COS. Most COS are still being developed with an HIC perspective and as such, still reflect the priorities from HIC perspectives. Only four COS have been initiated from LMICs. Public participation is still low and is poorly documented. There is, therefore, a need to explore how to enable initiation of COS development from a range of LMIC settings, how to ensure prioritisation of COS that better reflects the burden of disease in these contexts and how to improve public participation from LMICs.

    Data availability statement

    All data relevant to the study are included in the article or uploaded as online supplemental information.

    Ethics statements

    Patient consent for publication

    Acknowledgments

    This work is also published with the permission of the Director of KEMRI.

    References

    1. ↵
      1. Heneghan C,
      2. Goldacre B,
      3. Mahtani KR
      . Why clinical trial outcomes fail to translate into benefits for patients. Trials 2017;18:122. doi:10.1186/s13063-017-1870-2pmid:http://www.ncbi.nlm.nih.gov/pubmed/28288676
      OpenUrlCrossRefPubMed
    2. ↵
      1. Duffy JMN,
      2. Ziebland S,
      3. von Dadelszen P, et al
      . Tackling poorly selected, collected, and reported outcomes in obstetrics and gynecology research. Am J Obstet Gynecol 2019;220:71.e1–71.e4.doi:10.1016/j.ajog.2018.09.023pmid:http://www.ncbi.nlm.nih.gov/pubmed/30273584
      OpenUrlPubMed
    3. ↵
      1. Sinha IP,
      2. Williamson PR,
      3. Smyth RL
      . Outcomes in clinical trials of inhaled corticosteroids for children with asthma are narrowly focussed on short term disease activity. PLoS One 2009;4:e6276. doi:10.1371/journal.pone.0006276pmid:http://www.ncbi.nlm.nih.gov/pubmed/19609438
      OpenUrlCrossRefPubMed
    4. ↵
      1. Tong A,
      2. Crowe S,
      3. Gill JS, et al
      . Clinicians' and researchers' perspectives on establishing and implementing core outcomes in haemodialysis: semistructured interview study. BMJ Open 2018;8:e021198. doi:10.1136/bmjopen-2017-021198pmid:http://www.ncbi.nlm.nih.gov/pubmed/29678992
      OpenUrlAbstract/FREE Full Text
    5. ↵
      1. Clarke M
      . Standardising outcomes for clinical trials and systematic reviews. Trials 2007;8:1–3.doi:10.1186/1745-6215-8-39
      OpenUrlCrossRefPubMed
    6. ↵
      1. Chalmers I,
      2. Glasziou P
      . Avoidable waste in the production and reporting of research evidence. Lancet 2009;374:86–9.doi:10.1016/S0140-6736(09)60329-9pmid:http://www.ncbi.nlm.nih.gov/pubmed/19525005
      OpenUrlCrossRefPubMedWeb of Science
    7. ↵
      1. Williamson PR,
      2. Altman DG,
      3. Blazeby JM, et al
      . Developing core outcome sets for clinical trials: issues to consider. Trials 2012;13:132. doi:10.1186/1745-6215-13-132pmid:http://www.ncbi.nlm.nih.gov/pubmed/22867278
      OpenUrlCrossRefPubMed
    8. ↵
      1. Chevance A,
      2. Tran VT,
      3. Ravaud P
      . Improving the generalizability and credibility of core outcome sets (Coss) by a large and international participation of diverse stakeholders. J Clin Epidemiol 2020;125:1–8.doi:10.1016/j.jclinepi.2020.01.004
      OpenUrlPubMed
    9. ↵
      1. Hughes KL,
      2. Kirkham JJ,
      3. Clarke M, et al
      . Assessing the impact of a research funder's recommendation to consider core outcome sets. PLoS One 2019;14:e0222418. doi:10.1371/journal.pone.0222418pmid:http://www.ncbi.nlm.nih.gov/pubmed/31518375
      OpenUrlPubMed
    10. ↵
      1. COMET Initiative
      . COS endorsments, 2020. Available: http://www.comet-initiative.org/COSEndorsement [Accessed 26 Sep 2020].
    11. ↵
      1. Williamson PR,
      2. Altman DG,
      3. Bagley H, et al
      . The comet Handbook: version 1.0. Trials 2017;18:280. doi:10.1186/s13063-017-1978-4pmid:http://www.ncbi.nlm.nih.gov/pubmed/28681707
      OpenUrlCrossRefPubMed
    12. ↵
      1. Webbe JWH,
      2. Duffy JMN,
      3. Afonso E, et al
      . Core outcomes in neonatology: development of a core outcome set for neonatal research. Arch Dis Child Fetal Neonatal Ed 2020;105:fetalneonatal-2019-317501. doi:10.1136/archdischild-2019-317501pmid:http://www.ncbi.nlm.nih.gov/pubmed/31732683
      OpenUrlPubMed
    13. ↵
      1. COMET Initiative
      . COMET_database n.d. Available: http://www.comet-initiative.org/Studies [Accessed 10 Feb 2020].
    14. ↵
      1. Gargon E,
      2. Gurung B,
      3. Medley N, et al
      . Choosing important health outcomes for comparative effectiveness research: a systematic review. PLoS One 2014;9:e99111. doi:10.1371/journal.pone.0099111pmid:http://www.ncbi.nlm.nih.gov/pubmed/24932522
      OpenUrlCrossRefPubMed
    15. ↵
      1. Gorst SL,
      2. Gargon E,
      3. Clarke M, et al
      . Choosing important health outcomes for comparative effectiveness research: an updated review and user survey. PLoS One 2016;11:1–12.doi:10.1371/journal.pone.0146444
      OpenUrlCrossRefPubMed
    16. ↵
      1. Davis K,
      2. Gorst SL,
      3. Harman N, et al
      . Choosing important health outcomes for comparative effectiveness research: an updated systematic review and involvement of low and middle income countries. PLoS One 2018;13:1–14.doi:10.1371/journal.pone.0190695pmid:http://www.ncbi.nlm.nih.gov/pubmed/29438429
      OpenUrlCrossRefPubMed
    17. ↵
      1. Gargon E,
      2. Gorst SL,
      3. Williamson PR
      . Choosing important health outcomes for comparative effectiveness research: 5th annual update to a systematic review of core outcome sets for research. PLoS One 2019;14:1–18.doi:10.1371/journal.pone.0225980pmid:http://www.ncbi.nlm.nih.gov/pubmed/31830081
      OpenUrlCrossRefPubMed
    18. ↵
      1. Kirkham JJ,
      2. Davis K,
      3. Altman DG, et al
      . Core outcome Set-STAndards for development: the COS-STAD recommendations. PLoS Med 2017;14:e1002447. doi:10.1371/journal.pmed.1002447pmid:http://www.ncbi.nlm.nih.gov/pubmed/29145404
      OpenUrlPubMed
    19. ↵
      1. Gargon E,
      2. Gorst SL,
      3. Williamson PR
      . Choosing important health outcomes for comparative effectiveness research: 5th annual update to a systematic review of core outcome sets for research. PLoS One 2019;14:e0225980. doi:10.1371/journal.pone.0225980pmid:http://www.ncbi.nlm.nih.gov/pubmed/31830081
      OpenUrlPubMed
    20. ↵
      1. Sinha IP,
      2. Smyth RL,
      3. Williamson PR
      . Using the Delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLoS Med 2011;8:e1000393. doi:10.1371/journal.pmed.1000393pmid:http://www.ncbi.nlm.nih.gov/pubmed/21283604
      OpenUrlCrossRefPubMed
    21. ↵
      1. Turnbull AE,
      2. Dinglas VD,
      3. Friedman LA, et al
      . A survey of Delphi panelists after core outcome set development revealed positive feedback and methods to facilitate panel member participation. J Clin Epidemiol 2018;102:99–106.doi:10.1016/j.jclinepi.2018.06.007pmid:http://www.ncbi.nlm.nih.gov/pubmed/29966731
      OpenUrlPubMed
    22. ↵
      1. Williamson P,
      2. Altman D,
      3. Blazeby J, et al
      . Driving up the quality and relevance of research through the use of agreed core outcomes. J Health Serv Res Policy 2012;17:1–2.doi:10.1258/jhsrp.2011.011131pmid:http://www.ncbi.nlm.nih.gov/pubmed/22294719
      OpenUrlCrossRefPubMed
    23. ↵
      1. OECD
      . Development assistance Committee list of official development assistance recipients 2013. Available: http://www.oecd.org/dac/stats/49483614.pdf
    24. ↵
      1. Gorst SL,
      2. Gargon E,
      3. Clarke M, et al
      . Choosing important health outcomes for comparative effectiveness research: an updated review and identification of gaps. PLoS One 2016;11:1–14.doi:10.1371/journal.pone.0168403pmid:http://www.ncbi.nlm.nih.gov/pubmed/27973622
      OpenUrlCrossRefPubMed
    25. ↵
      1. Gargon E,
      2. Gorst SL,
      3. Harman NL, et al
      . Choosing important health outcomes for comparative effectiveness research: 4th annual update to a systematic review of core outcome sets for research. PLoS One 2018;13:1–15.doi:10.1371/journal.pone.0209869pmid:http://www.ncbi.nlm.nih.gov/pubmed/30592741
      OpenUrlCrossRefPubMed
    26. ↵
      1. Institute of Health Metrics and Evaluation
      . Global burden of disease 2020. Available: https://vizhub.healthdata.org/gbd-compare/ [Accessed 1 Dec 2020].
    27. ↵
      1. Murugupillai R,
      2. Ranganathan SS,
      3. Wanigasinghe J, et al
      . Development of outcome criteria to measure effectiveness of antiepileptic therapy in children. Epilepsy Behav 2018;80:56–60.doi:10.1016/j.yebeh.2017.12.011pmid:http://www.ncbi.nlm.nih.gov/pubmed/29414559
      OpenUrlPubMed
    28. ↵
      1. Xue Z,
      2. Sun J,
      3. Li T, et al
      . How to evaluate the clinical outcome of joint-preserving treatment for osteonecrosis of the femoral head: development of a core outcome set. J Orthop Surg Res 2019;14:317. doi:10.1186/s13018-019-1364-xpmid:http://www.ncbi.nlm.nih.gov/pubmed/31597557
      OpenUrlPubMed
    29. ↵
      1. Moher D,
      2. Liberati A,
      3. Tetzlaff J, et al
      . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009;6:e1000097. doi:10.1371/journal.pmed.1000097pmid:http://www.ncbi.nlm.nih.gov/pubmed/19621072
      OpenUrlCrossRefPubMed
    30. ↵
      1. Harbin Consensus Conference Workshop Group, Conference Chairs,
      2. Legro RS, et al
      . Improving the reporting of clinical trials of infertility treatments (imprint): modifying the CONSORT statement†‡. Hum Reprod 2014;29:2075–82.doi:10.1093/humrep/deu218pmid:http://www.ncbi.nlm.nih.gov/pubmed/25217611
      OpenUrlCrossRefPubMed
    31. ↵
      1. Cross H
      . A Delphi consensus on criteria for contraindications, assessment indicators and expected outcomes related to tibialis posterior transfer surgery. Int J Lepr Other Mycobact Dis 2005;73:13–21.doi:10.1489/1544-581X(2005)73%5B13:ADCOCF%5D2.0.CO;2pmid:http://www.ncbi.nlm.nih.gov/pubmed/15898834
      OpenUrlPubMed
    32. ↵
      1. Moorthy V,
      2. Reed Z,
      3. Smith PG, et al
      . Measurement of malaria vaccine efficacy in phase III trials: report of a who consultation. Vaccine 2007;25:5115–23.doi:10.1016/j.vaccine.2007.01.085pmid:http://www.ncbi.nlm.nih.gov/pubmed/17577487
      OpenUrlCrossRefPubMed
    33. ↵
      1. van 't Hooft J,
      2. Duffy JMN,
      3. Daly M, et al
      . A core outcome set for evaluation of interventions to prevent preterm birth. Obstet Gynecol 2016;127:49–58.doi:10.1097/AOG.0000000000001195pmid:http://www.ncbi.nlm.nih.gov/pubmed/26646133
      OpenUrlCrossRefPubMed
    34. ↵
      1. Karas J,
      2. Ashkenazi S,
      3. Guarino A, et al
      . A core outcome set for clinical trials in acute diarrhoea. Arch Dis Child 2015;100:359–63.doi:10.1136/archdischild-2014-307403pmid:http://www.ncbi.nlm.nih.gov/pubmed/25414251
      OpenUrlAbstract/FREE Full Text
    35. ↵
      1. Steutel NF,
      2. Benninga MA,
      3. Langendam MW, et al
      . Developing a core outcome set for infant colic for primary, secondary and tertiary care settings: a prospective study. BMJ Open 2017;7:1–11.doi:10.1136/bmjopen-2016-015418pmid:http://www.ncbi.nlm.nih.gov/pubmed/28554931
      OpenUrlCrossRefPubMed
    36. ↵
      1. Kuizenga-Wessel S,
      2. Steutel NF,
      3. Benninga MA, et al
      . Development of a core outcome set for clinical trials in childhood constipation: a study using a Delphi technique. BMJ Paediatr Open 2017;1:e000017. doi:10.1136/bmjpo-2017-000017pmid:http://www.ncbi.nlm.nih.gov/pubmed/29637094
      OpenUrlPubMed
    37. ↵
      1. Harman NL,
      2. Wilding JPH,
      3. Curry D, et al
      . Selecting core outcomes for randomised effectiveness trials in type 2 diabetes (SCORE-IT): a patient and healthcare professional consensus on a core outcome set for type 2 diabetes. BMJ Open Diabetes Res Care 2019;7:e000700. doi:10.1136/bmjdrc-2019-000700pmid:http://www.ncbi.nlm.nih.gov/pubmed/31908789
      OpenUrlAbstract/FREE Full Text
    38. ↵
      1. Miller AB,
      2. Hoogstraten B,
      3. Staquet M, et al
      . Reporting results of cancer treatment. Cancer 1981;47:207–14.doi:10.1002/1097-0142(19810101)47:1<207::AID-CNCR2820470134>3.0.CO;2-6pmid:http://www.ncbi.nlm.nih.gov/pubmed/7459811
      OpenUrlCrossRefPubMedWeb of Science
    39. ↵
      1. van der Heijde D,
      2. Bellamy N,
      3. Calin A, et al
      . Preliminary core sets for endpoints in ankylosing spondylitis. assessments in ankylosing spondylitis Working group. J Rheumatol 1997;24:2225–9.pmid:http://www.ncbi.nlm.nih.gov/pubmed/9375888
      OpenUrlPubMedWeb of Science
    40. ↵
      1. Vos T,
      2. Lim SS,
      3. Abbafati C, et al
      . Global burden of 369 diseases and injuries in 204 countries and territories, 1990-2019: a systematic analysis for the global burden of disease study 2019. Lancet 2020;396:1204–22.doi:10.1016/S0140-6736(20)30925-9pmid:http://www.ncbi.nlm.nih.gov/pubmed/33069326
      OpenUrlCrossRefPubMed
    41. ↵
      1. Gargon E,
      2. Williamson PR,
      3. Young B
      . Improving core outcome set development: qualitative interviews with developers provided pointers to inform guidance. J Clin Epidemiol 2017;86:140–52.doi:10.1016/j.jclinepi.2017.04.024pmid:http://www.ncbi.nlm.nih.gov/pubmed/28495644
      OpenUrlPubMed
    42. ↵
      1. Ong WL,
      2. Schouwenburg MG,
      3. van Bommel ACM, et al
      . A standard set of value-based patient-centered outcomes for breast cancer: the International Consortium for health outcomes measurement (ICHOM) initiative. JAMA Oncol 2017;3:677–85.doi:10.1001/jamaoncol.2016.4851pmid:http://www.ncbi.nlm.nih.gov/pubmed/28033439
      OpenUrlPubMed
    43. ↵
      1. Bloom G,
      2. Standing H,
      3. Lloyd R
      . Markets, information asymmetry and health care: towards new social contracts. Soc Sci Med 2008;66:2076–87.doi:10.1016/j.socscimed.2008.01.034pmid:http://www.ncbi.nlm.nih.gov/pubmed/18316147
      OpenUrlCrossRefPubMed
    44. ↵
      1. Young B,
      2. Bagley H
      . Including patients in core outcome set development: issues to consider based on three workshops with around 100 international delegates. Res Involv Engagem 2016;2:25.doi:10.1186/s40900-016-0039-6pmid:http://www.ncbi.nlm.nih.gov/pubmed/29507761
      OpenUrlPubMed
    45. ↵
      1. Hall DA,
      2. Smith H,
      3. Hibbert A, et al
      . The COMiT'ID study: developing core outcome domains sets for clinical trials of Sound-, Psychology-, and Pharmacology-Based interventions for chronic subjective tinnitus in adults. Trends Hear 2018;22:1–16.doi:10.1177/2331216518814384pmid:http://www.ncbi.nlm.nih.gov/pubmed/30488765
      OpenUrlCrossRefPubMed

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Footnotes

    • Twitter @jkarumb1, @GorstSarah

    • Contributors Study conception, design and planning of the study were by JK, PRW, DG, BY, SLG, EG as this was part of JK’s PhD. Data aggregation, analysis and writing the original draft were by JK. JK screened all title and abstracts supported by PRW and SLG who carried out double screening of these articles. Full-text articles were screened by JK and any uncertainties were discussed with the team. PRW is fully responsible for the overall content for the work and the conduct of the study, had access to the data, and controlled the decision to publish. All authors were involved with reviewing and editing the manuscript.

    • Funding JK has been supported to undertake this work as part of a PhD studentship from the University of Liverpool within the MRC/NIHR Trials Methodology Research Partnership.

    • Competing interests None declared.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.