Article Text

Original research
Systematic review of international Delphi surveys for core outcome set development: representation of international patients
  1. Alice Lee1,
  2. Anna Davies2,
  3. Amber E Young3,4
  1. 1 Academic Foundation Doctor, Department of Surgery and Cancer, Imperial College London, London, UK
  2. 2 Senior Research Fellow, Centre for Academic Child Health, University of Bristol, Bristol, UK
  3. 3 Consultant Paediatric Anaesthetist and Lead Children's Burns Research Centre, University Hospitals Bristol NHS Foundation Trust, Bristol, UK
  4. 4 Senior Research Fellow, Bristol Centre for Surgical Research, Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK
  1. Correspondence to Dr Amber E Young; amber.young1{at}


Objectives A core outcome set (COS) describes a minimum set of outcomes to be reported by all clinical trials of one healthcare condition. Delphi surveys are frequently used to achieve consensus on core outcomes. International input is important to achieve global COS uptake. We aimed to investigate participant representation in international Delphi surveys, with reference to the inclusion of patients and participants from low and middle income countries as stakeholders (LMICs).

Design Systematic review.

Data sources EMBASE, Medline, Web of Science, COMET database and hand-searching.

Eligibility criteria Protocols and studies describing Delphi surveys used to develop an international COS for trial reporting, published between 1 January 2017 and 6 June 2019.

Data extraction and synthesis Delphi participants were grouped as patients or healthcare professionals (HCPs). Participants were considered international if their country of origin was different to that of the first or senior author. Data extraction included participant numbers, country of origin, country income group and whether Delphi surveys were translated. We analysed the impact these factors had on outcome prioritisation.

Results Of 90 included studies, 69% (n=62) were completed and 31% (n=28) were protocols. Studies recruited more HCPs than patients (median 60 (IQR 30–113) vs 30 (IQR 14–66) participants, respectively). A higher percentage of HCPs was international compared with patients (57% (IQR 37–78) vs 20% (IQR 0–68)). Only 31% (n=28) studies recruited participants from LMICs. Regarding recruitment from LMICs, patients were under-represented (16% studies; n=8) compared with HCPs (22%; n=28). Few (7%; n=6) studies translated Delphi surveys. Only 3% studies (n=3) analysed Delphi responses by geographical location; all found differences in outcome prioritisation.

Conclusions There is a disproportionately lower inclusion of international patients, compared with HCPs, in COS-development Delphi surveys, particularly within LMICs. Future international Delphi surveys should consider exploring for geographical and income-based differences in outcome prioritisation.

PROSPERO registration number CRD42019138519.

  • Delphi
  • core outcome set
  • patient and public involvement
  • low and middle income countries
  • international

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:

Statistics from

Supplementary materials


  • Twitter @AliceEALee

  • Contributors AY devised the initial review question and provided critical review of the search strategy, data collection process and manuscript. AL designed the search strategy and was the primary researcher involved in screening search results, data extraction and drafting of the manuscript. AD provided critical review of the search strategy and acted as a second researcher for purposes of screening and data extraction, as well as critically reviewing the manuscript.

  • Funding This research (AL) was funded by The Scar Free Foundation. The Scar Free Foundation is the only medical research charity focused on scarring with the mission to achieve scar free healing within a generation. This study was also supported by the NIHR Biomedical Research Centre at University Hospitals Bristol and Weston NHS Foundation Trust and the University of Bristol. The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.

  • Map disclaimer The depiction of boundaries on this map does not imply the expression of any opinion whatsoever on the part of BMJ (or any member of its group) concerning the legal status of any country, territory, jurisdiction or area or of its authorities. This map is provided without any warranty of any kind, either express or implied.

  • Competing interests None declared.

  • Patient consent for publication Not required.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data availability statement Data are available in the supplementary material. Requests for data not included in the article or supplementary material can be made to the corresponding author.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.