You are here

Article Text

Article menu
XML
Research methods
Original research
Systematic review of international Delphi surveys for core outcome set development: representation of international patients
  1. Alice Lee1,
  2. Anna Davies2,
  3. Amber E Young3,4
  1. 1 Academic Foundation Doctor, Department of Surgery and Cancer, Imperial College London, London, UK
  2. 2 Senior Research Fellow, Centre for Academic Child Health, University of Bristol, Bristol, UK
  3. 3 Consultant Paediatric Anaesthetist and Lead Children's Burns Research Centre, University Hospitals Bristol NHS Foundation Trust, Bristol, UK
  4. 4 Senior Research Fellow, Bristol Centre for Surgical Research, Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK
  1. Correspondence to Dr Amber E Young; amber.young1{at}nhs.net

Abstract

Objectives A core outcome set (COS) describes a minimum set of outcomes to be reported by all clinical trials of one healthcare condition. Delphi surveys are frequently used to achieve consensus on core outcomes. International input is important to achieve global COS uptake. We aimed to investigate participant representation in international Delphi surveys, with reference to the inclusion of patients and participants from low and middle income countries as stakeholders (LMICs).

Design Systematic review.

Data sources EMBASE, Medline, Web of Science, COMET database and hand-searching.

Eligibility criteria Protocols and studies describing Delphi surveys used to develop an international COS for trial reporting, published between 1 January 2017 and 6 June 2019.

Data extraction and synthesis Delphi participants were grouped as patients or healthcare professionals (HCPs). Participants were considered international if their country of origin was different to that of the first or senior author. Data extraction included participant numbers, country of origin, country income group and whether Delphi surveys were translated. We analysed the impact these factors had on outcome prioritisation.

Results Of 90 included studies, 69% (n=62) were completed and 31% (n=28) were protocols. Studies recruited more HCPs than patients (median 60 (IQR 30–113) vs 30 (IQR 14–66) participants, respectively). A higher percentage of HCPs was international compared with patients (57% (IQR 37–78) vs 20% (IQR 0–68)). Only 31% (n=28) studies recruited participants from LMICs. Regarding recruitment from LMICs, patients were under-represented (16% studies; n=8) compared with HCPs (22%; n=28). Few (7%; n=6) studies translated Delphi surveys. Only 3% studies (n=3) analysed Delphi responses by geographical location; all found differences in outcome prioritisation.

Conclusions There is a disproportionately lower inclusion of international patients, compared with HCPs, in COS-development Delphi surveys, particularly within LMICs. Future international Delphi surveys should consider exploring for geographical and income-based differences in outcome prioritisation.

PROSPERO registration number CRD42019138519.

  • Delphi
  • core outcome set
  • patient and public involvement
  • low and middle income countries
  • international
http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

http://dx.doi.org/10.1136/bmjopen-2020-040223

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • We conducted a comprehensive systematic search of the literature including three search engines, the Core Outcome Measures in Effectiveness Trials database and hand-searching.

    • This is the first review to present stakeholder-level data on the country of origin and income status of Delphi survey participants for core outcome set studies.

    • Study reporting on participant demographics was inconsistent and some data were not in a compatible format for data analysis.

    • The literature search was time limited (January 2017 to June 2019) and restricted to English language articles.

    Introduction

    A core outcome set (COS) is a minimum group of outcomes to be reported in all trials of a specific health condition. Development is undertaken to reduce the heterogeneity of outcome reporting across trials and to enable study results to be compared and combined to inform best medical practice. Core outcomes are identified scientifically by stakeholders as being the most important in determining the effects of an intervention or treatment in one healthcare condition.1–3 Consensus in outcome prioritisation among stakeholders in COS development is often achieved using Delphi surveys.2 4 A diverse group of stakeholders is recommended to be recruited, including patients, healthcare professionals (HCPs), trialists, regulators, industry representatives, policymakers, researchers and the public.2 The Delphi process comprises iterative rounds of surveys in which the importance of outcomes is rated.2 After each round, the participants’ individual responses, and those of other stakeholders, are fed back in an anonymised manner, so that they can be reconsidered before the next round in an aim to achieve consensus. The Delphi method is advantageous because it incorporates the views of various stakeholder groups and can be conducted electronically (‘e-Delphi’) to facilitate international participation.1 2 4

    International participation in Delphi surveys is important for the COS to be applicable in global healthcare settings, and because widespread uptake of the COS will facilitate the future synthesis of trial evidence on an international scale.2 An increasing number of COS developers are including international participants.5–93 A recent survey reported that approximately 50% of published COS projects from the last 5 years included participants from two or more countries.92 Despite this increase in international stakeholder participation, there is no agreement on how study methodology should be adapted to facilitate such input. The Core Outcome Measures in Effectiveness Trials (COMET) Handbook highlights the logistical and organisational challenges of international COS development projects as well as issues regarding generalisability of small international participant numbers.2 The effect of income group and/or geographical location of Delphi participants on core outcome prioritisation has not been systematically investigated, despite international variation in healthcare resources, biomedical beliefs and burden of disease.5 73 81

    Using a systematic review of the literature, this study aims to explore participant representation in international Delphi surveys used for COS development. Part of our analysis will explore how COS projects undertaking international Delphi surveys evaluate the impact of participants from countries from different World Bank income groups on prioritising the importance of outcomes.

    Methods

    This systematic review adheres to a prespecified protocol and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement.94 The protocol for this review was registered on PROSPERO (available from: https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42019138519).

    Identification of studies

    Study eligibility

    COS development studies/protocols were included where international participants took part in a Delphi survey to prioritise outcomes for a COS, published between 1 January 2017 and 6 June 2019. The inclusion and exclusion criteria are shown in table 1. This time period was chosen to reflect the 2017 publication of the COMET Handbook2 to maximise study homogeneity. Protocol studies were included for methodological detail if equivalent full-length studies were not available, because the review focused on evaluating methodology, not study outcomes. Full-text and protocol studies describing the same international Delphi survey were merged as one reference.

    View this table:
    Table 1

    Study inclusion and exclusion criteria

    Types of participants

    For inclusion in the review, at least one participant in the Delphi survey had to be international, defined as a different country of origin to the first or senior author. Participants were included if they completed at least round 1 of the Delphi survey. Those who registered but did not participate were not included.

    Types of interventions

    Included studies used an international Delphi survey for prioritisation of outcomes at some point in the consensus process (including modified Delphi surveys and mixed methods approaches). Any study intervention and comparator were acceptable.

    Types of outcome

    Included studies used an international Delphi survey to prioritise any outcomes for a health condition or intervention to form a COS.

    Search strategy

    A search string was developed to identify relevant papers. These included key search terms and their synonyms (eg, COS, international, Delphi) and relevant medical subject headings. The search string for Ovid MEDLINE is shown in online supplemental appendix 1 and was adapted for different databases (Ovid EMBASE, Web of Science).

    Supplemental material

    Study selection process

    Search results were compiled using Mendeley (V.1.19.4). Citations were deduplicated using Mendeley software and manually. Article screening was undertaken by one researcher (AL) against prespecified inclusion and exclusion criteria (table 1) in two stages (title and abstract and full text). For both stages, a second researcher (AD) independently assessed 20% of the screening results. Inter-rater reliability between researchers was assessed with Cohen’s kappa.95 If discrepancies in article selection could not be resolved, a third researcher (AY) was consulted.

    Quality assessment

    A risk of bias assessment was not undertaken because the review aimed to assess study methodology and not the effect of study interventions. There is currently no risk of bias assessment tool for COS development or Delphi surveys, and tools for assessing risk of bias in trials or observational studies are not applicable to these reviewed studies.

    Data extraction

    Data were extracted using a piloted data extraction form (Microsoft Excel) developed for the purpose of the review. Data were extracted under the following domains: study details (year of publication, full text/protocol study and COMET disease category), participant numbers, international status of participants, participant geographical location and income group, the effect of these on prioritisation and whether the Delphi survey was translated.

    Number of Delphi participants overall and per stakeholder group

    participants were grouped into two stakeholder groups: patients and representatives (carers and representatives from patient organisations) and HCPs (medical professionals, trialists, regulators, industry representatives, policymakers and researchers). Data were extracted on number of participants (total and per stakeholder group). We recorded number of participants per study based first on the number of participants from the Delphi round which included both patients and HCPs. If both stakeholder groups were included throughout or the study only included either patients or HCPs from the outset, we extracted number of participants from the Delphi round with the largest number of participants.

    International status of participants and effect on outcome prioritisation

    Participants were categorised as international if their country of origin was not the same as either the first or senior author of the study. Other demographic data included participants’ countries of origin and the World Bank world regions and World Bank income groups represented by these countries. All study texts were scrutinised for any description of analysis of Delphi responses by geographical location or income status, and if so, the outcome of this analysis.

    Delphi survey translation

    Studies that recruited participants from non-English-speaking countries were scrutinised for any description of survey translation, and if so, details of the translated languages and method of translation.

    Data synthesis

    Data from individual studies were tabulated by one author (AL). A second researcher (AD) independently extracted data from 20% of included studies. Inter-rater reliability was assessed with Cohen’s kappa.95 If discrepancies in data extraction could not be resolved, a third researcher (AY) was consulted. Quantitative, non-parametric data were analysed using Microsoft Excel to calculate medians and IQRs for number of participants, percentage of international participants and number of countries of origin, World Bank world regions and World Bank income groups. Categorical data were described narratively, including participant countries of origin, survey translation and the outcomes of analysis of responses by geographical location or income status.

    Patient and public involvement

    No patients were involved in this review of previously published data.

    Results

    Identification of studies

    The electronic search identified 529 non-duplicate citations, of which 90 were included in the final data set (figure 1). Cohen’s kappa demonstrated very good (0.81) and good (0.71) agreement between researchers performing screening at the abstract and full-text stages, respectively.96 Of the 90 included studies (online supplemental appendix 3), 69% (n=62) were completed and 31% (n=28) were study protocols. The greatest number of Delphi studies were published from the UK (42%; n=38; online supplemental appendix 3) and the three most frequent COMET disease categories were pregnancy and childbirth (14%, n=13), gastroenterology (9%, n=8) and orthopaedics and trauma (9%, n=8); online supplemental appendix 4.

    Supplemental material

    Supplemental material

    Figure 1
    Figure 1

    Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram. COS, core outcome set.

    Number of Delphi participants overall and per stakeholder group

    The median number of participants per study was 100 (table 2). Most studies (77%; n=69) included both patients and HCPs. Of the 23% of studies (n=21) with only one stakeholder group, 95% (n=20) recruited only international HCPs. Of all studies, 70% (n=48) recruited international participants in both stakeholder groups, 19% (n=13) recruited into only one stakeholder group (92% of which recruited only HCPs) and in 12% (n=8), the status of stakeholder groups was unclear. Studies recruited two times as many HCPs as patients (median 60 vs 30 participants).

    View this table:
    Table 2

    Demographic data for the overall cohort, patients and HCPs

    International status of participants and effect on outcome prioritisation

    The median percentage of international participants per study was 52%. Studies recruited three times more international HCPs than international patients (57 vs 20%). The total number of countries represented across the included studies was 95 for HCPs and 46 for patients. Within these studies, the median number of countries represented in each Delphi was 11 for HCPs and 2 for patients.

    Participants were recruited from every World Bank world region for both stakeholder groups. HCPs represented two times as many World Bank world regions as patients (four vs two regions). The most frequent countries of participant origin were the USA and UK for HCPs and patients (figure 2). The most frequent World Bank world regions reported for both HCPs and patients were Europe and Central Asia, followed by North America and East Asia and the Pacific (figure 3). Compared with HCPs, fewer studies recruited patients from certain world regions (Sub-Saharan Africa, Latin America and the Caribbean and East Asia and the Pacific) (figure 3). Only 4% studies (n=2) recruited patients from Sub-Saharan Africa when compared with studies recruiting HCPs (13%; n=10).

    Figure 2
    Figure 2

    World map showing participant countries of origin for (A) patients and (B) HCPs. The colour gradient represents the percentage of studies which recruited patients and HCPs from each country. For patients, this ranged from 2% (light blue) to 28% (dark blue) and for HCPs this ranged from 1% (light blue) to 39% (dark blue). HCPs, healthcare professionals.

    Figure 3
    Figure 3

    Distribution of (A) World Bank world regions and (B) World Bank income groups of participants by overall cohort and stakeholder group. HCPs, healthcare professionals.

    Most studies recruited participants from high-income countries (48%; n=43), followed by high–middle income (28%; n=25), lower middle income (16%; n=15) and low income countries (9%; n=8; figure 3). HCPs were recruited from two times as many World Bank income groups as patients (two vs one groups). Less than half as many studies recruited patients from low-income (2%, n=1) and lower middle income countries (6%; n=3) when compared with studies recruiting HCPs (5%, n=4% and 14%, n=11, respectively).

    A minority of studies (4%, n=4) analysed, or stated an intention to analyse, the Delphi survey responses by participants’ geographical location, either by country54 97 or continent.73 Some differences in outcome prioritisation were minor, not affecting the final COS.54 97 Park et al presented results from round 1 of their Delphi survey on patient-reported outcomes (PROs) for adult myositis.54 They found that, unlike participants from the USA and South Korea, Swedish participants rated ‘impact on household activity’ less favourably, although this outcome was still retained for the second Delphi round. Sautenet et al reported consensus outcomes for kidney transplantation among patients, caregivers and HCPs.97 They found that patients/caregivers from certain countries ranked ‘depression’ or ‘cognition’ as less important and that ‘skin cancer’ received greater prioritisation in countries with public campaigns for prevention. Since none of the aforementioned outcomes were ranked in the ‘top eight’ of either patients or HCPs, these differences did not affect the final COS. Van Rijssen et al 73 reported consensus PROs for pancreatic cancer among European, North American and Asian participants. In this study, the outcomes in the final COS would have been different if the responses were analysed by continent rather than as a whole cohort. In comparison to the whole cohort, European participants reached consensus ‘in’ on three additional outcomes, American participants reached consensus ‘in’ for one additional outcome, but did not reach consensus for two outcomes included in the final COS, and Asian participants did not reach consensus on any PROs included in the final COS. A fourth study (protocol) stated an intention to analyse Delphi survey results by ‘language and cultural variation’.17 None of the included studies analysed Delphi responses by income status.

    Delphi survey translation

    Delphi surveys were translated in a minority of studies that recruited participants from countries in which English was not the first language (16%, n=6/38). This included five studies with both stakeholder groups and one with patients only. Three studies translated the Delphi survey into all languages spoken by the patients54 73 83 and a protocol study mentioned translating the Delphi survey ‘as required’.98 One study translated the Delphi survey from English into Dutch only, despite recruiting patients from many other countries.27 One study translated the Delphi survey from English to Italian but did not list countries of origin for patients.99 Some studies (n=14) excluded non-English-speaking participants, despite recruiting patients and representatives from many countries where English is not the first language.8 13 18 19 21 22 25 36 55 56 59 60 69 79

    Discussion

    The findings of this systematic review demonstrate that most international Delphi surveys for COS development recruited both HCPs and patients. Patients were recruited in fewer numbers and were less likely to be international and especially from LMICs. A minority of studies altered Delphi survey language, despite recruiting participants from several countries with non-English first languages. Importantly, few studies analysed Delphi responses by geographical location of participants, but those that did found differences in outcome prioritisation.

    These findings echo those of annual systematic reviews of COS development studies, demonstrating lower overall recruitment of Delphi participants from LMICs.5 6 Our review adds new, stakeholder-level information, which has identified a disproportionately lower recruitment of patients (overall and particularly within LMICs). There are various possible explanations for this, including lower English language proficiency, reduced internet access (required for e-Delphi surveys and online recruitment), differing biomedical beliefs and lack of resources and/or time for voluntary research participation.100–102 Development of contextual methods to effectively engage international patients from LMICs is required.

    Lack of Delphi survey translation is an important barrier to participation, particularly affecting patients from LMICs. Most included studies did not adapt survey language for participants and many excluded non-English speakers. This could have introduced recruitment bias, particularly within LMICs. Researchers have expressed concern that Delphi survey translation could result in loss of comprehension or meaning.2 Some of the included studies have approached this problem by translating outcomes by native-speaking professionals only,98 using an online multilanguage interface103 or by discussing translated outcomes with patient research partners.54 Van Rijssen et al 73 used a method of forward and backward translation, that is, translated the survey from English to the required participant languages and then back into English. Discrepancies between the forward and backward translations were resolved by consensus between the translators and if necessary, the project team. This is the method recommended by the COMET handbook,2 WHO104 and MAPI Institute105 for providing equivalent conceptual, cultural and semantic meaning and should be encouraged as the first-line method after consideration of the costs involved.

    Increasing the international participation of Delphi surveys for COS development should be encouraged but raises considerations for study design. For example, how to ensure Delphi studies have appropriate and adequate international representation, the criteria to define this (geographically and/or using income status) and how to analyse international data. None of the included studies analysed Delphi responses by income status and few analysed Delphi responses by geographical location of participants. Those that did found differences in prioritisation between participants of different countries of origin54 60 and continents of origin.73 Importantly, in the study by Van Rijssen et al,73 which reported a core set of PROs for pancreatic cancer, participants from Europe, the USA and Asia did not reach the same consensus on the final COS, and Asian participants did not reach consensus on any PROs included in the final set. There is no clear explanation for these discrepancies. For COSs to have global uptake, it is important that the selected outcomes are representative of all international stakeholders. Performing subanalyses of Delphi responses by participant location and/or income status should be encouraged as a useful indication of applicability across different populations, and significant differences in prioritisation should be explored.

    Another important finding of this systematic review was inconsistent study reporting. In some studies, basic demographic information was unclear or not specified including which stakeholder groups were international and from which countries and world regions they were recruited. Many studies only listed the most frequent countries of origin, provided data in non-standard formats, for example, reporting participant numbers for the USA and Canada combined, or only provided geographic detail for the whole cohort instead of per stakeholder group. Some studies provided demographic data on invited participants, rather than those who had completed at least one round of the Delphi survey. As a result, we were unable to use many studies’ data in our analyses. Without adequate demographic information, it is difficult to interpret the applicability of COSs across different populations. Current reporting checklists for COS development projects, such as Core Outcome Set–STAndards for Reporting (COS-STAR),106 could be adapted to reflect international participation.

    The findings of this review should be interpreted in the context of its limitations. The HCP group consisted of various stakeholders with potentially differing views, including healthcare workers, trialists, regulators, industry representatives, policymakers and researchers. Not all studies included each of these constituent groups. For ease of comparison, and because the largest distinction in opinion was likely to be between HCPs and patients, all professionals were combined into one group. This approach has been used previously.107 We described patients as international if their country of origin was different to the affiliated country of either the first or last author of the study. Some authors had multiple affiliations, which may have reduced the apparent percentage of international participants. We used the World Bank world regions and income groups to categorise demographic information from participants, but some studies did not present their demographic data in a compatible format. Furthermore, we restricted our search to articles available in English and published within a recent time frame.

    Conclusion

    In conclusion, we have demonstrated that Delphi surveys used to develop COSs for clinical trials recruit fewer international patients than HCPs, particularly from LMICs. This could be contributed to by a lack of Delphi survey translation for non-English speakers. Few studies explored any geographical variation in responses; those that did found differences in outcome prioritisation. This review highlights complex issues that need further discussion, for example, how to define adequate international participation and how to analyse international data, including geographical discrepancies in outcome selection. Future studies should consider exploratory analyses of Delphi responses by geographical location or income status to assess applicability of the COS across populations. Study reporting was inconsistent and could be improved with a standardised checklist for international Delphi surveys.

    Supplemental material

    References

    1. ↵
      2. Williamson PR ,
      3. Altman DG ,
      4. Blazeby JM , et al
      . Developing core outcome sets for clinical trials: issues to consider. Trials 2012;13:1–8.doi:10.1186/1745-6215-13-132
      OpenUrlCrossRefPubMed
    2. ↵
      2. Williamson PR ,
      3. Altman DG ,
      4. Bagley H , et al
      . The comet Handbook: version 1.0. Trials 2017;18:1–50.doi:10.1186/s13063-017-1978-4
      OpenUrlCrossRef
    3. ↵
      2. Clarke M ,
      3. Williamson PR
      . Core outcome sets and systematic reviews. Syst Rev 2016;5:11. doi:10.1186/s13643-016-0188-6
    4. ↵
      2. Hall DA ,
      3. Smith H ,
      4. Heffernan E , et al
      . Recruiting and retaining participants in e-Delphi surveys for core outcome set development: evaluating the COMiT'ID study. PLoS One 2018;13:e0201378. doi:10.1371/journal.pone.0201378
    5. ↵
      2. Davis K ,
      3. Gorst SL ,
      4. Harman N , et al
      . Choosing important health outcomes for comparative effectiveness research: an updated systematic review and involvement of low and middle income countries. PLoS One 2018;13:e0190695. doi:10.1371/journal.pone.0190695 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29438429
      OpenUrlPubMed
    6. ↵
      2. Gargon E ,
      3. Gorst SL ,
      4. Harman NL , et al
      . Choosing important health outcomes for comparative effectiveness research: 4th annual update to a systematic review of core outcome sets for research. PLoS One 2018;13:e0209869:1–15. doi:10.1371/journal.pone.0209869 pmid:http://www.ncbi.nlm.nih.gov/pubmed/30592741
      OpenUrlPubMed
    7. ↵
      2. Haller G ,
      3. Bampoe S ,
      4. Cook T , et al
      . Systematic review and consensus definitions for the standardised endpoints in perioperative medicine initiative: clinical indicators. Br J Anaesth 2019;123:228–37.doi:10.1016/j.bja.2019.04.041
      OpenUrl
    8. ↵
      2. Butler DP ,
      3. De la Torre A ,
      4. Borschel GH , et al
      . An international collaborative standardizing patient-centered outcome measures in pediatric facial palsy. JAMA Facial Plast Surg 2019;21:351–8.doi:10.1001/jamafacial.2019.0224 pmid:http://www.ncbi.nlm.nih.gov/pubmed/31070677
      OpenUrlPubMed
    9. ↵
      2. Byrne M ,
      3. O’Connell A ,
      4. Egan AM , et al
      . A core outcomes set for clinical trials of interventions for young adults with type 1 diabetes: an international, multi-perspective Delphi consensus study. Trials 2017;18:602. doi:10.1186/s13063-017-2364-y
    10. ↵
      2. Cho Y ,
      3. Sautenet B ,
      4. Rangan G , et al
      . Standardised outcomes in Nephrology—Polycystic kidney disease (SONG-PKD): study protocol for establishing a core outcome set in polycystic kidney disease. Trials 2017;18:560. doi:10.1186/s13063-017-2298-4
    11. ↵
      2. Dadouch R ,
      3. Faheim M ,
      4. Juando-Prats C , et al
      . Development of a core outcome set for studies on obesity in pregnant patients (COSSOPP): a study protocol. Trials 2018;19:665. doi:10.1186/s13063-018-3029-1
    12. ↵
      2. Danese S ,
      3. Bonovas S ,
      4. Lopez A , et al
      . Identification of Endpoints for Development of Antifibrosis Drugs for Treatment of Crohn’s Disease. Gastroenterology 2018;155:76–87.doi:10.1053/j.gastro.2018.03.032
      OpenUrl
    13. ↵
      2. Dorman SL ,
      3. Shelton JA ,
      4. Stevenson RA , et al
      . Management of medial humeral epicondyle fractures in children: a structured review protocol for a systematic review of the literature and identification of a core outcome set using a Delphi survey. Trials 2018;19:119. doi:10.1186/s13063-018-2472-3
    14. ↵
      2. Dos Santos F ,
      3. Drymiotou S ,
      4. Antequera Martin A , et al
      . Development of a core outcome set for trials on induction of labour: an international multistakeholder Delphi study. BJOG: Int J Obstet Gy 2018;125:1673–80.doi:10.1111/1471-0528.15397
      OpenUrl
    15. ↵
      2. Callis Duffin K ,
      3. Merola JF ,
      4. Christensen R , et al
      . Identifying a core domain set to assess psoriasis in clinical trials. JAMA Dermatology 2018;154:1137–44.doi:10.1001/jamadermatol.2018.1165
      OpenUrl
    16. ↵
      2. Duffy JMN ,
      3. Bhattacharya S ,
      4. Curtis C , et al
      . A protocol developing, disseminating and implementing a core outcome set for infertility. Human Reproduction Open 2018;2018:hoy007.doi:10.1093/hropen/hoy007
      OpenUrl
    17. ↵
      2. Alkhaffaf B ,
      3. Bruce I ,
      4. Glenny AM , et al
      . The GASTROS study: standardising outcome reporting in gastric cancer surgery research. Eur J Cancer 2017;72:S134. doi:10.1016/S0959-8049(17)30517-8
    18. ↵
      2. Evangelidis N ,
      3. Tong A ,
      4. Manns B , et al
      . Developing a set of core outcomes for trials in hemodialysis: an international Delphi survey. American Journal of Kidney Diseases 2017;70:464–75.doi:10.1053/j.ajkd.2016.11.029
      OpenUrlCrossRefPubMed
    19. ↵
      2. Fish R ,
      3. Sanders C ,
      4. Adams R , et al
      . A core outcome set for clinical trials of chemoradiotherapy interventions for anal cancer (CORMAC): a patient and health-care professional consensus. The Lancet Gastroenterology & Hepatology 2018;3:865–73.doi:10.1016/S2468-1253(18)30264-4
      OpenUrl
    20. ↵
      2. Foust-Wright C ,
      3. Wissig S ,
      4. Stowell C , et al
      . Development of a core set of outcome measures for OAB treatment. Int Urogynecol J 2017;28:1785–93.doi:10.1007/s00192-017-3481-6
      OpenUrl
    21. ↵
      2. Gagnier JJ ,
      3. Page MJ ,
      4. Huang H , et al
      . Creation of a core outcome set for clinical trials of people with shoulder pain: a study protocol. Trials 2017;18:336. doi:10.1186/s13063-017-2054-9
    22. ↵
      2. Hall DA ,
      3. Smith H ,
      4. Hibbert A , et al
      . The COMiT’ID Study: Developing Core Outcome Domains Sets for Clinical Trials of Sound-, Psychology-, and Pharmacology-Based Interventions for Chronic Subjective Tinnitus in Adults. Trends in Hearing 2018;22:233121651881438.doi:10.1177/2331216518814384
      OpenUrl
    23. ↵
      2. Harman NL ,
      3. Wilding J ,
      4. Curry D , et al
      . Selecting core outcomes for randomised effectiveness trials in type 2 diabetes (SCORE-IT): study protocol for the development of a core outcome set. Trials 2018;19:427. doi:10.1186/s13063-018-2805-2
    24. ↵
      2. Haywood K ,
      3. Whitehead L ,
      4. Nadkarni VM , et al
      . COSCA (core outcome set for cardiac arrest) in adults: an Advisory statement from the International liaison Committee on resuscitation. Resuscitation 2018;127:147–63.doi:10.1016/j.resuscitation.2018.03.022
      OpenUrlPubMed
    25. ↵
      2. Healy P ,
      3. Gordijn S ,
      4. Ganzevoort W , et al
      . Core outcome set for growth restriction: deVeloping endpoints (COSGROVE). Trials 2018;19:451. doi:10.1186/s13063-018-2819-9
    26. ↵
      2. Hodgson CL ,
      3. Turnbull AE ,
      4. Iwashyna TJ , et al
      . Core domains in evaluating patient outcomes after acute respiratory failure: international multidisciplinary clinician consultation. Phys Ther 2017;97:168–74.doi:10.2522/ptj.20160196
      OpenUrl
    27. ↵
      2. Horbach SER ,
      3. van der Horst CMAM ,
      4. Blei F , et al
      . Development of an international core outcome set for peripheral vascular malformations: the OVAMA project. Br J Dermatol 2018;178:473–81.doi:10.1111/bjd.16029
      OpenUrl
    28. ↵
      2. Allin BSR ,
      3. Bradnock T ,
      4. Kenny S , et al
      . NETS 1HD study: development of a Hirschsprung’s disease core outcome set. Arch Dis Child 2017;102:1143–51.doi:10.1136/archdischild-2017-312901
      OpenUrlAbstract/FREE Full Text
    29. ↵
      2. Kampstra NA ,
      3. Grutters JC ,
      4. van Beek FT , et al
      . First patient-centred set of outcomes for pulmonary sarcoidosis: a multicentre initiative. BMJ Open Respiratory Research 2019;6:e000394. doi:10.1136/bmjresp-2018-000394
    30. ↵
      2. Kelly A ,
      3. Tong A ,
      4. Tymms K , et al
      . Outcome Measures in Rheumatology - Interventions for medication Adherence (OMERACT-Adherence) Core Domain Set for Trials of Interventions for Medication Adherence in Rheumatology: 5 Phase Study Protocol. Trials 2018;19:204. doi:10.1186/s13063-018-2565-z
    31. ↵
      2. Kenny KP ,
      3. Day PF ,
      4. Sharif MO , et al
      . What are the important outcomes in traumatic dental injuries? an international approach to the development of a core outcome set. Dental Traumatology 2018;34:4–11.doi:10.1111/edt.12367
      OpenUrl
    32. ↵
      2. Khalil A ,
      3. Beune I ,
      4. Hecher K , et al
      . Consensus definition and essential reporting parameters of selective fetal growth restriction in twin pregnancy: a Delphi procedure. Ultrasound Obstet Gynecol 2019;53:47–54.doi:10.1002/uog.19013
      OpenUrl
    33. ↵
      2. Kim AH ,
      3. Roberts C ,
      4. Feagan BG , et al
      . Developing a standard set of patient-centred outcomes for inflammatory bowel Disease—an international, Cross-disciplinary consensus. Journal of Crohn's and Colitis 2018;12:408–18.doi:10.1093/ecco-jcc/jjx161
      OpenUrl
    34. ↵
      2. Knaapen M ,
      3. Hall NJ ,
      4. van der Lee JH , et al
      . Establishing a core outcome set for treatment of uncomplicated appendicitis in children: study protocol for an international Delphi survey. BMJ Open 2019;9:e028861. doi:10.1136/bmjopen-2018-028861 pmid:http://www.ncbi.nlm.nih.gov/pubmed/31123006
      OpenUrlAbstract/FREE Full Text
    35. ↵
      2. Leo DG ,
      3. Leong WY ,
      4. Gambling T , et al
      . The outcomes of Perthes’ disease of the hip: a study protocol for the development of a core outcome set. Trials 2018;19:374. doi:10.1186/s13063-018-2695-3
    36. ↵
      2. Ma C ,
      3. Panaccione R ,
      4. Fedorak RN , et al
      . Development of a core outcome set for clinical trials in inflammatory bowel disease: study protocol for a systematic review of the literature and identification of a core outcome set using a Delphi survey. BMJ Open 2017;7:e016146. doi:10.1136/bmjopen-2017-016146
    37. ↵
      2. Mackie SL ,
      3. Twohig H ,
      4. Neill LM , et al
      . The OMERACT core domain set for outcome measures for clinical trials in polymyalgia rheumatica. J Rheumatol 2017;44:1515–21.doi:10.3899/jrheum.161109
      OpenUrlAbstract/FREE Full Text
    38. ↵
      2. MacLennan S ,
      3. Williamson PR ,
      4. Bekema H , et al
      . A core outcome set for localised prostate cancer effectiveness trials. BJU Int 2017;120:E64–79.doi:10.1111/bju.13854
      OpenUrl
    39. ↵
      2. Sherratt F ,
      3. Bagley H ,
      4. Stones S , et al
      . Improving core outcome set development for children and young people: learning from a case study in acute appendicitis and consultation with an international group of children and young people. J Evid Based Med 2019;12:21.
    40. ↵
      2. Manera KE ,
      3. Tong A ,
      4. Craig JC , et al
      . Standardized outcomes in Nephrology—Peritoneal dialysis (SONG-PD): study protocol for establishing a core outcome set in PD. Perit Dial Int 2017;37:639–47.doi:10.3747/pdi.2017.00022
      OpenUrlAbstract/FREE Full Text
    41. ↵
      2. Matvienko-Sikar K ,
      3. Byrne M ,
      4. Kelly C , et al
      . Development of an infant feeding core outcome set for childhood obesity interventions: study protocol. Trials 2017;18:463. doi:10.1186/s13063-017-2180-4
    42. ↵
      2. Maujean A ,
      3. Carroll L ,
      4. Curatolo M , et al
      . A core outcome set for clinical trials in whiplash-associated disorders (WAD): a study protocol. Trials 2018;19:635. doi:10.1186/s13063-018-3019-3
    43. ↵
      2. McIlroy DR ,
      3. Bellomo R ,
      4. Billings FT , et al
      . Systematic review and consensus definitions for the standardised endpoints in perioperative medicine (step) initiative: renal endpoints. Br J Anaesth 2018;121:1013–24.doi:10.1016/j.bja.2018.08.010
      OpenUrlPubMed
    44. ↵
      2. McPhee PG ,
      3. Benner JL ,
      4. Balemans ACJ , et al
      . Multimorbidity risk assessment in adolescents and adults with cerebral palsy: a protocol for establishing a core outcome set for clinical research and practice. Trials 2019;20:176. doi:10.1186/s13063-019-3265-z
    45. ↵
      2. Meher S ,
      3. Cuthbert A ,
      4. Kirkham JJ , et al
      . Core outcome sets for prevention and treatment of postpartum haemorrhage: an international Delphi consensus study. BJOG: Int J Obstet Gy 2019;126:83–93.doi:10.1111/1471-0528.15335
      OpenUrl
    46. ↵
      2. Milman N ,
      3. Boonen A ,
      4. Tugwell P , et al
      . Clinicians’ perspective on key domains in ANCA-associated vasculitis: a Delphi exercise. Scand J Rheumatol 2017;46:112–7.doi:10.1080/03009742.2016.1188980
      OpenUrl
    47. ↵
      2. Murugupillai R ,
      3. Ranganathan SS ,
      4. Wanigasinghe J , et al
      . Development of outcome criteria to measure effectiveness of antiepileptic therapy in children. Epilepsy & Behavior 2018;80:56–60.doi:10.1016/j.yebeh.2017.12.011
      OpenUrl
    48. ↵
      2. Nabbout R ,
      3. Auvin S ,
      4. Chiron C , et al
      . Development and content validation of a preliminary core set of patient- and caregiver-relevant outcomes for inclusion in a potential composite endpoint for Dravet syndrome. Epilepsy & Behavior 2018;78:232–42.doi:10.1016/j.yebeh.2017.08.029
      OpenUrl
    49. ↵
      2. Nijagal MA ,
      3. Wissig S ,
      4. Stowell C , et al
      . Standardized outcome measures for pregnancy and childbirth, an ICHOM proposal. BMC Health Serv Res 2018;18:953. doi:10.1186/s12913-018-3732-3
    50. ↵
      2. Balakrishnan K ,
      3. Sidell DR ,
      4. Bauman NM , et al
      . Outcome measures for pediatric laryngotracheal reconstruction: international consensus statement. Laryngoscope 2019;129:244–55.doi:10.1002/lary.27445
      OpenUrlPubMed
    51. ↵
      2. Obbarius A ,
      3. van Maasakkers L ,
      4. Baer L , et al
      . Standardization of health outcomes assessment for depression and anxiety: recommendations from the ICHOM depression and anxiety Working group. Qual Life Res 2017;26:3211–25.doi:10.1007/s11136-017-1659-5
      OpenUrl
    52. ↵
      2. Ong WL ,
      3. Schouwenburg MG ,
      4. van Bommel ACM , et al
      . A standard set of value-based patient-centered outcomes for breast cancer: the International Consortium for health outcomes measurement (ICHOM) initiative. JAMA Oncol 2017;3:677–85.doi:10.1001/jamaoncol.2016.4851 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28033439
      OpenUrlPubMed
    53. ↵
      2. Oude Voshaar MAH ,
      3. Das Gupta Z ,
      4. Bijlsma JWJ , et al
      . International Consortium for health outcome measurement set of outcomes that matter to people living with inflammatory arthritis: consensus from an international Working group. Arthritis Care Res 2019;71:1556–65.doi:10.1002/acr.23799 pmid:http://www.ncbi.nlm.nih.gov/pubmed/30358135
      OpenUrlPubMed
    54. ↵
      2. Park JK ,
      3. Mecoli CA ,
      4. Alexanderson H , et al
      . Advancing the development of patient-reported outcomes for adult myositis at OMERACT 2016: an international Delphi study. J Rheumatol 2017;44:1683–7.doi:10.3899/jrheum.161252
      OpenUrlAbstract/FREE Full Text
    55. ↵
      2. Perry H ,
      3. Duffy JMN ,
      4. Reed K , et al
      . Core outcome set for research studies evaluating treatments for twin-twin transfusion syndrome. Ultrasound Obstet Gynecol 2019;54:255–61.doi:10.1002/uog.20183 pmid:http://www.ncbi.nlm.nih.gov/pubmed/30520170
      OpenUrlPubMed
    56. ↵
      2. Prins JR ,
      3. Holvast F ,
      4. van ’t Hooft J , et al
      . Development of a core outcome set for immunomodulation in pregnancy (COSIMPREG): a protocol for a systematic review and Delphi study. BMJ Open 2018;8:e021619. doi:10.1136/bmjopen-2018-021619
    57. ↵
      2. Rankin A ,
      3. Cadogan CA ,
      4. ın Ryan C , et al
      . Core outcome set for trials aimed at improving the appropriateness of polypharmacy in older people in primary care. J Am Geriatr Soc 2018;66:1206–12.doi:10.1111/jgs.15245
      OpenUrl
    58. ↵
      2. Regardt M ,
      3. Mecoli CA ,
      4. Park JK , et al
      . OMERACT 2018 modified patient-reported outcome domain core set in the life impact area for adult idiopathic inflammatory myopathies. J Rheumatol 2019;46:1351–4.doi:10.3899/jrheum.181065
      OpenUrlAbstract/FREE Full Text
    59. ↵
      2. Rose L ,
      3. Agar M ,
      4. Burry LD , et al
      . Development of core outcome sets for effectiveness trials of interventions to prevent and/or treat delirium (Del-COrS): study protocol. BMJ Open 2017;7:e016371. doi:10.1136/bmjopen-2017-016371
    60. ↵
      2. Sautenet B ,
      3. Tong A ,
      4. Chapman J , et al
      . Developing a core outcome set in trials for adult kidney transplant recipients: an international Delphi survey. J Evid Based Med 2017;10:16.
      OpenUrl
    61. ↵
      2. Bampoe S ,
      3. Cook T ,
      4. Fleisher L , et al
      . Clinical indicators for reporting the effectiveness of patient quality and safety-related interventions: a protocol of a systematic review and Delphi consensus process as part of the International standardised endpoints for perioperative medicine initiative (step). BMJ Open 2018;8:e023427. doi:10.1136/bmjopen-2018-023427
    62. ↵
      2. Schaap T ,
      3. Bloemenkamp K ,
      4. Deneux-Tharaux C , et al
      . Defining definitions: a Delphi study to develop a core outcome set for conditions of severe maternal morbidity. BJOG: Int J Obstet Gy 2019;126:394–401.doi:10.1111/1471-0528.14833
      OpenUrl
    63. ↵
      2. Schlessinger DI ,
      3. Iyengar S ,
      4. Yanes AF , et al
      . Development of a core outcome set for clinical trials in basal cell carcinoma: study protocol for a systematic review of the literature and identification of a core outcome set using a Delphi survey. Trials 2017;18:490. doi:10.1186/s13063-017-2244-5
    64. ↵
      2. Shorter GW ,
      3. Heather N ,
      4. Bray JW , et al
      . The ‘Outcome Reporting in Brief Intervention Trials: Alcohol’ (ORBITAL) framework: protocol to determine a core outcome set for efficacy and effectiveness trials of alcohol screening and brief intervention. Trials 2017;18:611. doi:10.1186/s13063-017-2335-3
    65. ↵
      2. Smith SM ,
      3. Wallace E ,
      4. Salisbury C , et al
      . A core outcome set for multimorbidity research (COSmm). Ann Fam Med 2018;16:132–8.doi:10.1370/afm.2178
      OpenUrlAbstract/FREE Full Text
    66. ↵
      2. Somner JEA ,
      3. Ismail R ,
      4. Froud R , et al
      . Consensus generation of a minimum set of outcome measures for auditing glaucoma surgery outcomes—a Delphi exercise. Graefes Arch Clin Exp Ophthalmol 2018;256:2407–11.doi:10.1007/s00417-018-4140-4
      OpenUrl
    67. ↵
      2. Steutel NF ,
      3. Benninga MA ,
      4. Langendam MW , et al
      . Developing a core outcome set for infant colic for primary, secondary and tertiary care settings: a prospective study. BMJ Open 2017;7:e015418. doi:10.1136/bmjopen-2016-015418
    68. ↵
      2. Tallouzi MO ,
      3. Mathers JM ,
      4. Moore DJ , et al
      . COSUMO: study protocol for the development of a core outcome set for efficacy and effectiveness trials in posterior segment-involving uveitis. Trials 2017;18:576. doi:10.1186/s13063-017-2294-8
    69. ↵
      2. Thorlacius L ,
      3. Ingram JR ,
      4. Villumsen B , et al
      . A core domain set for hidradenitis suppurativa trial outcomes: an international Delphi process. Br J Dermatol 2018;179:642–50.doi:10.1111/bjd.16672
      OpenUrl
    70. ↵
      2. Turnbull AE ,
      3. Sepulveda KA ,
      4. Dinglas VD , et al
      . Core domains for clinical research in acute respiratory failure survivors: an international modified Delphi consensus study. Crit Care Med 2017;45:1001–10.doi:10.1097/CCM.0000000000002435 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28375853
      OpenUrlPubMed
    71. ↵
      2. Van den Bussche K ,
      3. Kottner J ,
      4. Beele H , et al
      . Core outcome domains in incontinence-associated dermatitis research. J Adv Nurs 2018;74:1605–17.doi:10.1111/jan.13562
      OpenUrl
    72. ↵
      2. Benstoem C ,
      3. Moza A ,
      4. Meybohm P , et al
      . A core outcome set for adult cardiac surgery trials: a consensus study. PLoS One 2017;12:e0186772. doi:10.1371/journal.pone.0186772
    73. ↵
      2. van Rijssen LB ,
      3. Gerritsen A ,
      4. Henselmans I , et al
      . Core set of patient-reported outcomes in pancreatic cancer (COPRAC). Ann Surg 2019;270:158–64.doi:10.1097/SLA.0000000000002633
      OpenUrl
    74. ↵
      2. Van Tol R ,
      3. Melenhorst J ,
      4. Dirksen C , et al
      . Developing a core outcome set in clinical trials for haemorrhoidal disease: an international Delphi study. Color Dis 2017;19:138.
      OpenUrl
    75. ↵
      2. Verberne WR ,
      3. Das-Gupta Z ,
      4. Allegretti AS , et al
      . Development of an international standard set of value-based outcome measures for patients with chronic kidney disease: a report of the International Consortium for health outcomes measurement (ICHOM) CKD Working group. Am J Kidney Dis 2019;73:372–84.doi:10.1053/j.ajkd.2018.10.007 pmid:http://www.ncbi.nlm.nih.gov/pubmed/30579710
      OpenUrlPubMed
    76. ↵
      2. Viau-Lapointe J ,
      3. D'Souza R ,
      4. Rose L , et al
      . Development of a core outcome set for research on critically ill obstetric patients: a study protocol. Obstet Med 2018;11:132–6.doi:10.1177/1753495X18772996 pmid:http://www.ncbi.nlm.nih.gov/pubmed/30214479
      OpenUrlPubMed
    77. ↵
      2. Wallace SJ ,
      3. Worrall L ,
      4. Rose T , et al
      . Which treatment outcomes are most important to aphasia clinicians and managers? an international e-Delphi consensus study. Aphasiology 2017;31:643–73.doi:10.1080/02687038.2016.1186265
      OpenUrl
    78. ↵
      2. Webbe J ,
      3. Gale C
      . Core outcomes in neonatology: a core outcome set based on routinely collected data. J Evid Based Med 2017;10:12.
    79. ↵
      2. Wuytack F ,
      3. Gutke A ,
      4. Stuge B , et al
      . Protocol for the development of a core outcome set for pelvic girdle pain, including methods for measuring the outcomes: the PGP-COS study. BMC Med Res Methodol 2018;18:158. doi:10.1186/s12874-018-0624-5
    80. ↵
      2. Young A ,
      3. Brookes S ,
      4. Rumsey N , et al
      . Agreement on what to measure in randomised controlled trials in burn care: study protocol for the development of a core outcome set. BMJ Open 2017;7:e017267. doi:10.1136/bmjopen-2017-017267
    81. ↵
      2. Zack R ,
      3. Okunade O ,
      4. Olson E , et al
      . Improving hypertension outcome measurement in low- and middle-income countries. Hypertension 2019;73:990–7.doi:10.1161/HYPERTENSIONAHA.118.11916
      OpenUrl
    82. ↵
      2. Zerillo JA ,
      3. Schouwenburg MG ,
      4. van Bommel ACM , et al
      . An international collaborative standardizing a comprehensive patient-centered outcomes measurement set for colorectal cancer. JAMA Oncol 2017;3:686–94.doi:10.1001/jamaoncol.2017.0417
      OpenUrl
    83. ↵
      2. Beuscart J-B ,
      3. Knol W ,
      4. Cullinan S , et al
      . International core outcome set for clinical trials of medication review in multi-morbid older patients with polypharmacy. BMC Med 2018;16.doi:10.1186/s12916-018-1007-9
    84. ↵
      2. de Roos P ,
      3. Bloem BR ,
      4. Kelley TA , et al
      . A Consensus Set of Outcomes for Parkinson’s Disease from the International Consortium for Health Outcomes Measurement. J Parkinsons Dis 2017;7:533–43.doi:10.3233/JPD-161055
      OpenUrl
    85. ↵
      2. Allori AC ,
      3. Kelley T ,
      4. Meara JG , et al
      . A standard set of outcome measures for the comprehensive appraisal of cleft care. Cleft Palate Craniofac J 2017;54:540–54.doi:10.1597/15-292 pmid:http://www.ncbi.nlm.nih.gov/pubmed/27223626
      OpenUrlPubMed
    86. ↵
      2. Egan AM ,
      3. Galjaard S ,
      4. Maresh MJA , et al
      . A core outcome set for studies evaluating the effectiveness of prepregnancy care for women with pregestational diabetes. Diabetologia 2017;60:1190–6.doi:10.1007/s00125-017-4277-4
      OpenUrl
    87. ↵
      2. Singh JA ,
      3. Dohm M ,
      4. Choong PF
      . Consensus on draft OMERACT core domains for clinical trials of total joint replacement outcome by orthopaedic surgeons: a report from the International consensus on outcome measures in TJR trials (I-COMiTT) group. BMC Musculoskelet Disord 2017;18:45. doi:10.1186/s12891-017-1409-4
    88. ↵
      2. Spargo M ,
      3. Ryan C ,
      4. Downey D , et al
      . Development of a core outcome set for trials investigating the long-term management of bronchiectasis. Chron Respir Dis 2019;16:147997231880416. doi:10.1177/1479972318804167
    89. ↵
      2. Agha RA ,
      3. Pidgeon TE ,
      4. Borrelli MR , et al
      . Validated outcomes in the grafting of autologous fat to the breast. Plast Reconstr Surg 2018;141:633e–8.doi:10.1097/PRS.0000000000004273
      OpenUrl
    90. ↵
      2. Hopkins C ,
      3. Hettige R ,
      4. Soni-Jaiswal A , et al
      . Chronic rhinosinusitis outcome measures (chrome), developing a core outcome set for trials of interventions in chronic rhinosinusitis. Rhin 2018;56:22–32.doi:10.4193/Rhin17.247
      OpenUrl
    91. ↵
      2. Iorio A ,
      3. Skinner MW ,
      4. Clearfield E , et al
      . Core outcome set for gene therapy in haemophilia: results of the coreHEM multistakeholder project. Haemophilia 2018;24:e167–72.doi:10.1111/hae.13504
      OpenUrl
    92. ↵
      2. Biggane AM ,
      3. Brading L ,
      4. Ravaud P , et al
      . Survey indicated that core outcome set development is increasingly including patients, being conducted internationally and using Delphi surveys. Trials 2018;19:113. doi:10.1186/s13063-018-2493-y
    93. ↵
      2. Bogdanet D ,
      3. Reddin C ,
      4. Macken E , et al
      . Follow-up at 1 year and beyond of women with gestational diabetes treated with insulin and/or oral glucose-lowering agents: a core outcome set using a Delphi survey. Diabetologia 2019;62:2007–16.doi:10.1007/s00125-019-4935-9
      OpenUrlPubMed
    94. ↵
      2. Liberati A ,
      3. Altman DG ,
      4. Tetzlaff J , et al
      . The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol 2009;62:e1–34.doi:10.1016/j.jclinepi.2009.06.006
      OpenUrlCrossRefPubMed
    95. ↵
      2. Cohen J
      . Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychol Bull 1968;70:213–20.doi:10.1037/h0026256
      OpenUrlCrossRefPubMedWeb of Science
    96. ↵
      2. Altman D
      . Practical statistics for medical research. Chapman & Hall, 1991.
    97. ↵
      2. Sautenet B ,
      3. Tong A ,
      4. Manera KE , et al
      . Developing consensus-based priority outcome domains for trials in kidney transplantation. Transplantation 2017;101:1875–86.doi:10.1097/TP.0000000000001776
      OpenUrlPubMed
    98. ↵
      2. Knaapen M ,
      3. Hall NJ ,
      4. van der Lee JH , et al
      . Protocol for the development of a global core outcome set for reporting treatment of uncomplicated appendicitis in children. J Evid Based Med 2019;12:15.
      OpenUrl
    99. ↵
      2. Smith TO ,
      3. Hawker GA ,
      4. Hunter DJ , et al
      . The OMERACT-OARSI core domain set for measurement in clinical trials of hip and/or knee osteoarthritis. J Rheumatol 2019;46:981–9.doi:10.3899/jrheum.181194
      OpenUrlAbstract/FREE Full Text
    100. ↵
      2. Bernier R ,
      3. Halpin E ,
      4. Staffa SJ , et al
      . Inclusion of non-English-speaking patients in research: a single institution experience. Pediatric Anesthesia 2018;28:415–20.doi:10.1111/pan.13363
      OpenUrl
    101. ↵
      2. Puchalski Ritchie LM ,
      3. Khan S ,
      4. Moore JE , et al
      . Low- and middle-income countries face many common barriers to implementation of maternal health evidence products. J Clin Epidemiol 2016;76:229–37.doi:10.1016/j.jclinepi.2016.02.017
      OpenUrlPubMed
    102. ↵
      2. Cook N ,
      3. Siddiqi N ,
      4. Twiddy M , et al
      . Patient and public involvement in health research in low and middle-income countries: a systematic review. BMJ Open 2019;9:e026514. doi:10.1136/bmjopen-2018-026514
    103. ↵
      2. Beuscart JB ,
      3. Dalleur O ,
      4. Boland B , et al
      . Development of a core outcome set for medication review in older patients with multimorbidity and polypharmacy: a study protocol. Clin Interv Aging 2017;12:1379–89.doi:10.2147/CIA.S135481
      OpenUrl
    104. ↵
      1. Process of translation and adaptation of instruments [Internet]
      . World health organisation 2020.
    105. ↵
      1. MAPI Institute
      . Linguistic validation manual for health outcome assessments. 2nd ed. MAPI Research Trust, 2012.
    106. ↵
      2. Kirkham JJ ,
      3. Gorst S ,
      4. Altman DG , et al
      . Core outcome set-standards for reporting: the COS-STAR statement. J Evid Based Med 2017;10:7.
    107. ↵
      2. Jones JE ,
      3. Jones LL ,
      4. Keeley TJH , et al
      . A review of patient and carer participation and the use of qualitative research in the development of core outcome sets. PLoS One 2017;12:e0172937:1–18. doi:10.1371/journal.pone.0172937 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28301485
      OpenUrlPubMed

    Supplementary materials

    Footnotes

    • Twitter @AliceEALee

    • Contributors AY devised the initial review question and provided critical review of the search strategy, data collection process and manuscript. AL designed the search strategy and was the primary researcher involved in screening search results, data extraction and drafting of the manuscript. AD provided critical review of the search strategy and acted as a second researcher for purposes of screening and data extraction, as well as critically reviewing the manuscript.

    • Funding This research (AL) was funded by The Scar Free Foundation. The Scar Free Foundation is the only medical research charity focused on scarring with the mission to achieve scar free healing within a generation. This study was also supported by the NIHR Biomedical Research Centre at University Hospitals Bristol and Weston NHS Foundation Trust and the University of Bristol. The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.

    • Map disclaimer The depiction of boundaries on this map does not imply the expression of any opinion whatsoever on the part of BMJ (or any member of its group) concerning the legal status of any country, territory, jurisdiction or area or of its authorities. This map is provided without any warranty of any kind, either express or implied.

    • Competing interests None declared.

    • Patient consent for publication Not required.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data availability statement Data are available in the supplementary material. Requests for data not included in the article or supplementary material can be made to the corresponding author.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.