Article Text

Original research
What items should be included in an early warning score for remote assessment of suspected COVID-19? qualitative and Delphi study
  1. Trisha Greenhalgh1,
  2. Paul Thompson2,
  3. Sietse Weiringa1,
  4. Ana Luisa Neves3,
  5. Laiba Husain1,
  6. Merlin Dunlop4,
  7. Alexander Rushforth1,
  8. David Nunan1,
  9. Simon de Lusignan1,
  10. Brendan Delaney3
  1. 1Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK
  2. 2Department of Experimental Psychology, University of Oxford, Oxford, UK
  3. 3Institute of Global Health Innovation, Imperial College London, London, UK
  4. 4Ardens Ltd, Oxford, UK
  1. Correspondence to Trisha Greenhalgh; trish.greenhalgh{at}


Background To develop items for an early warning score (RECAP: REmote COVID-19 Assessment in Primary Care) for patients with suspected COVID-19 who need escalation to next level of care.

Methods The study was based in UK primary healthcare. The mixed-methods design included rapid review, Delphi panel, interviews, focus groups and software development. Participants were 112 primary care clinicians and 50 patients recovered from COVID-19, recruited through social media, patient groups and snowballing. Using rapid literature review, we identified signs and symptoms which are commoner in severe COVID-19. Building a preliminary set of items from these, we ran four rounds of an online Delphi panel with 72 clinicians, the last incorporating fictional vignettes, collating data on R software. We refined the items iteratively in response to quantitative and qualitative feedback. Items in the penultimate round were checked against narrative interviews with 50 COVID-19 patients. We required, for each item, at least 80% clinician agreement on relevance, wording and cut-off values, and that the item addressed issues and concerns raised by patients. In focus groups, 40 clinicians suggested further refinements and discussed workability of the instrument in relation to local resources and care pathways. This informed design of an electronic template for primary care systems.

Results The prevalidation RECAP-V0 comprises a red flag alert box and 10 assessment items: pulse, shortness of breath or respiratory rate, trajectory of breathlessness, pulse oximeter reading (with brief exercise test if appropriate) or symptoms suggestive of hypoxia, temperature or fever symptoms, duration of symptoms, muscle aches, new confusion, shielded list and known risk factors for poor outcome. It is not yet known how sensitive or specific it is.

Conclusions Items on RECAP-V0 align strongly with published evidence, clinical judgement and patient experience. The validation phase of this study is ongoing.

Trial registration number NCT04435041.

  • infectious diseases
  • health informatics
  • virology
  • covid-19

This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See:

Statistics from

Supplementary materials


  • Twitter @trishgreenhalgh, @Lusignan_S

  • Contributors TG conceptualised the study and oversaw all work packages described in the paper. She led or contributed to the various rapid reviews, recruited participants, led data analysis and summarising and wrote the paper. PT designed and led the operational and quantitative aspects of the Delphi study and the quantitative anlyses. SW and ALN recruited participants, cofacilitated the focus groups and participated in data analysis and writing up. LH provided assistance with all aspects of the research. MD designed the templates for putting RECAP items on general practice electronic records and participated in focus groups. AR interviewed patient participants and extracted relevant data for this study. DN led rapid reviews on predictive value of signs and symptoms. SdeL and BD contributed to data interpretation and analysis, especially in relation to coding and linkage of electronic data, and design of the validation study. All authors checked and approved the final manuscript.

  • Funding The RECAP study was funded primarily from a UK Research and Innovation grant ‘Remote By Default’ under the Covid-19 Emergency Fund. Additional funding for some staff salaries came from National Institute for Health Research (BRC-1215-20008) and Wellcome Trust (WT104830MA). Development of the coding templates was part funded by the CommunityJameel Imperial College COVID-19 Excellence Fund.

  • Competing interests MD’s company makes templates for general practice record systems. All other authors declare no competing interests.

  • Patient and public involvement statement Several of the research team are clinicians who developed COVID-19 and thus bring lived experience of the condition. The study was over seen by an independent advisory group with a lay chair and a separate patient advisory group consisting of people from diverse social and ethnic backgrounds who have had COVID-19; this group had input to the study design. The experience of patients was also captured systematically through patient interviews as an integral component of the research, and this led to amendments to the RECAP items. Some patient participants were recruited by snow balling from other patient participants and some from a patient support group. Results will be disseminated to all patient participants by emailing or posting a lay summary.

  • Patient consent for publication Not required.

  • Ethics approvals and consent to participate Ethical approval for the patient interviews was obtained from East Midlands—LeicesterCentral Research Ethics Committee (IRAS Project ID: 283196; REC ref 20/EM0128) on 4 May 2020. Ethical approval for the validation study (including holding the focus groups and building and installing the software on primary care record systems) was obtained from Greater Manchester Research Ethics Committee on 20 May 2020 (IRAS Project ID: 283024; REC ref 20/NW/0266) and subsequent amendments. In accordance with our ethics approvals, all patient participants gave written informed consent and all clinician focus group participants gave verbal consent which was recorded on the video. We sought advice from the chair of our local NHS Research Ethics Committee who considered that the online Delphi panel did not require formal ethical approval; consent from these participants was obtained by email.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data availability statement Additional data, including an extensive summary of qualitative findings, are supplied as appendices. Researchers seeking further raw data should contact the corresponding author.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.