Article Text


Comparing clinical quality indicators for asthma management in children with outcome measures used in randomised controlled trials: a protocol
  1. Miew Keen Choong1,
  2. Guy Tsafnat1,
  3. Peter Hibbert2,3,
  4. William B Runciman2,3,4,
  5. Enrico Coiera1
  1. 1Centre for Health Informatics, Australian Institute of Health Innovation, Faculty of Medicine and Health Sciences, Macquarie University, Sydney, New South Wales, Australia
  2. 2Centre for Health Resilience and Implementation Science, Australian Institute of Health Innovation, Faculty of Medicine and Health Sciences, Macquarie University, Sydney, New South Wales, Australia
  3. 3Centre for Population Health Research, University of South Australia, Adelaide, South Australia, Australia
  4. 4Australian Patient Safety Foundation, Adelaide, South Australia, Australia
  1. Correspondence to Dr Miew Keen Choong; miewkeen.choong{at}


Introduction Clinical quality indicators are necessary to monitor the performance of healthcare services. The development of indicators should, wherever possible, be based on research evidence to minimise the risk of bias which may be introduced during their development, because of logistic, ethical or financial constraints alone. The development of automated methods to identify the evidence base for candidate indicators should improve the process of indicator development. The objective of this study is to explore the relationship between clinical quality indicators for asthma management in children with outcome and process measurements extracted from randomised controlled clinical trial reports.

Methods and analysis National-level indicators for asthma management in children will be extracted from the National Quality Measures Clearinghouse (NQMC) database and the National Institute for Health and Care Excellence (NICE) quality standards. Outcome measures will be extracted from published English language randomised controlled trial (RCT) reports for asthma management in children aged below 12 years. The two sets of measures will be compared to assess any overlap. The study will provide insights into the relationship between clinical quality indicators and measurements in RCTs. This study will also yield a list of measurements used in RCTs for asthma management in children, and will find RCT evidence for indicators used in practice.

Ethics and dissemination Ethical approval is not necessary because this study will not include patient data. Findings will be disseminated through peer-reviewed publications.


This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See:

Statistics from

Strengths and limitations of this study

  • Develop a novel method for comparing indicators with research evidence in clinical trial.

  • Provide a systematic and objective way to identify candidate indicators, identify research associated with current indicators, and identify potential inconsistencies between research evidence and clinical practice.

  • Generate a list of measurements used in randomised controlled trials of asthma management in children.


The drive to improve the quality and safety of healthcare has resulted in the proliferation of clinical quality indicators. Indicators may be used to assess the quality of care, or to identify and prioritise areas for improvement.1 ,2 They describe “as much about a system as possible in as few points as possible”.3

A good indicator should be important, relevant, valid, reliable, meaningful and understandable.3 ,4 In addition, indicators should be easy to collect,5 and the costs of doing so should not outweigh the potential benefits.6 The development of indicators can be complex and resource intensive. Indicators can be developed using deductive approaches (moving from target concepts to supporting data), inductive approaches (using data to identify concepts) or both.7 Deductive approaches are more common.7 The steps to developing indicators deductively are: (1) selecting a topic, (2) forming a review team, (3) researching the evidence and practice behind candidate indicators, (4) selecting indicators, (5) designing their specifications, (6) pilot testing their collection and (7) implementing them.2 ,8

In practice, clinical indicators can be vulnerable to bias and lack rigour8 because of (sometimes necessary) logistic, ethical or financial constraints during their development. To minimise these risks, the development of indicators should, wherever possible, be based on research evidence.2 ,9 Different grades of evidence are available, and randomised controlled trials (RCT) are generally regarded as one of the highest levels of evidence.10 ,11

Asthma is a chronic respiratory disease that affects about 300 million people globally, with an estimated 250 000 deaths annually.12 It is the most common chronic disease among children. There is currently no cure for asthma, but it can generally be managed and controlled with appropriate care (care regarded as being in line with clinical guidelines13–15). Many quality indicators have been developed for asthma;16–20 most are based on clinical practice guidelines (CPGs). For example, the National Institute for Health and Care Excellence (NICE) quality standards for asthma are based on the British Thoracic Society/Scottish Intercollegiate Guidelines Network (BTS/SIGN) clinical guideline 101: British guidelines on the management of asthma;13 and the asthma clinical performance measurement tool promoted by the Physician Consortium for Performance Improvement is based on the National Asthma Education and Prevention Program Expert Panel Report 2.21

To better assess the level of evidence behind currently recommended indicators, this protocol describes a study of the relationship between outcome measurements used in RCTs and national level clinical quality indicators in the UK and the USA, using asthma management in children as a case study. The study will compare outcome terms extracted from RCT reports with clinical quality indicators by assigning concepts from the Unified Medical Language System (UMLS) using MetaMap22 to both sets of terms. This allows terms from different sources to be standardised for easy aggregation and comparison. MetaMap has been used for many tasks, for example, information retrieval23 and text mining,24 ,25 and has been shown to perform well in mapping biomedical concepts to text compared with human labelling.26

Methods and analysis

The study will examine asthma management in children aged less than 12 years. There are four basic steps: (1) identification and extraction of indicators, (2) identification and extraction of outcome measurements from RCTs, (3) mapping both sets of metrics to underlying concepts as defined in the UMLS using MetaMap and (4) matching and comparing clinical quality indicators and RCT measurements, with an evaluation of the basis for any differences.

Identification and extraction of indicators

National-level clinical quality indicators for asthma management in children from the USA and the UK will be collected. The National Quality Measures Clearinghouse (NQMC) is a database on healthcare quality measures sponsored by the US Department of Health and Human Services—Agency for Healthcare Research and Quality (AHRQ). The NQMC will be searched using the term ‘asthma’ within the respiratory tract diseases condition (50 measures in asthma of the 243). The NICE quality standard for asthma (11 asthma quality standards) will also be included. NICE plays a major role in commissioning and producing guidelines and developing quality standards for Britain's National Health Service (NHS).

The following inclusion criteria will be used to select indicators:

  1. Any process or outcome indicator,

  2. Any indicator of pharmacological or non-pharmacological management,

  3. Applicable to children aged below 12 years,

  4. Developed for national-level assessment.

Candidate indicators identified in the search will be independently assessed against inclusion criteria by two appraisers. Disagreements will be resolved by consensus.

The included indicators will then be normalised as follows:

  1. Remove: Unit of analysis (eg, number of people/patients),

  2. Remove: Time points/frames (eg, more than 24 h, within 2 days) for a process to occur,

  3. Remove: The term “asthma” if it describes the characteristics of patients,

  4. Extract: Noun terms describing the unit of analysis (examples in table 1).

Table 1

Examples of classification of extracted terms

Table 2 shows examples of possible terms that may be extracted from the indicator sources.

Table 2

Examples of terms extracted from clinical indicators

Identification and extraction of outcomes from RCTs

Types of studies

Only RCTs will be included. Secondary/post hoc analysis/protocol of RCTs will be excluded.

Types of participants

Children with asthma aged below 12 years will be included.

Types of interventions

Studies that involved any type of medical or non-medical management of asthma will be included.

Search strategy

We will search PubMed for RCTs on asthma management for children aged below 12 years. Studies will be restricted to English language articles published in the past 10 years (2005–2014).

The following search terms will be used to retrieve PubMed articles:

  1. Asthma

  2. (randomised controlled trial[Publication Type] OR (randomised[Title/Abstract] AND controlled[Title/Abstract] AND trial[Title/Abstract]))

  3. #1 AND #2

  4. #3 AND “English”[Language]

The second search term above is the specific/narrow therapy filter for clinical queries. PubMed filters of Species (‘Humans’) and Ages (‘Infant: birth-23 months’, ‘Preschool Child: 2–5 years’, and ‘Child: 6–12 years’) will be applied.

Study selection

The titles and abstracts of the resulting references will be screened independently by two reviewers against the following inclusion criteria:

  1. Articles describing RCT, excluding secondary/post hoc analysis/protocol of RCTs;

  2. The participants of the trial include children with asthma aged below 12 years even if the trial also includes other participants;

  3. The trial's focus is on medical or non-medical management of asthma.

Any control arm is acceptable as well as any outcome measure. Disagreements will be resolved by consensus. Inter-rater agreement will be measured periodically on screening decisions and only one screener may be used for the remaining abstracts if agreement is high (ie, Cohen's κ>0.8).27

The full text of the potentially eligible studies will be retrieved and reassessed if the abstract does not contain enough information to make a decision.

Outcome measures extraction from clinical trials and ambiguities resolution

RCTs generally report using the PICO criteria:28 Population, Intervention, Comparison, and Outcome. Two independent annotators will be asked to independently extract the Population (ie, number of participants in the study) and Outcome (outcome names) from the full text of every included RCT. Outcome terms will be any phrase that describes a quantifiable measurement.

Outcome names identified by this process will be normalised using the following rules:

  1. Outcome deduplication: Similar outcome terms mentioned multiple times in an RCT will only be considered as a single occurrence.

  2. Generalisation relationships: The most specific term mentioned in an RCT will be preferred. For example, spirometry and peak flow measurement both describe lung function tests, but the latter is more specific.

  3. Ambiguities: Any ambiguities will be resolved by consultation with a paediatric respiratory physician, paediatric nurse or other relevant clinician.

Inter-rater agreement will be measured periodically on extraction decisions and only one extractor may be used for the remaining extractions if agreement is high (ie, Cohen's κ>0.8).

UMLS concept mapping using MetaMap

The list of terms extracted from indicator sources and the list of outcome measures taken from RCTs will next both be mapped into standardised terms (UMLS concepts using MetaMap) to facilitate aggregation and comparison. The UMLS Metathesaurus, from the National Library of Medicine, comprises over 1 million biomedical concepts from over 100 source vocabularies. MetaMap22 is a tool for mapping biomedical terms in free text to the UMLS Metathesaurus. MetaMap will be provided with extracted terms from clinical indicators and RCTs and will return a list of ranked mapping options. We will use all concepts identified by MetaMap in this study. If no UMLS concept is found, we will retain the extracted term. Table 3 shows examples of phrases mapped to UMLS terms. As shown in table 3, ‘exacerbation of asthma’ will be mapped to ‘asthma exacerbation’ and ‘PEF’ will be mapped to ‘peak expiratory flow’ even if they are expressed differently.

Table 3

Examples of mapping to UMLS terms using MetaMap

Extracted term ranking

UMLS concepts will be ranked according to the total number of RCT participants that they were applied to. In most cases, this is the number of patients randomised or for whom there was an intention to treat.

Mapping and comparing clinical quality indicators and RCT measurements

The RCT outcomes and practice indicators will then be compared using UMLS concepts by two annotators. Terms in both lists will be mapped to each other if one or more UMLS concepts associated with them match. The mappings will be agreed by consensus and inter-rater reliability will be measured. Advice will be sought from appropriate clinicians if consensus cannot be reached.

The congruence between the mapped outcomes from RCTs and those from clinical indicators will be measured. Measured outcomes in RCT that are not used as clinical indicators may imply wastage either because metrics with research evidence behind them are not being used as indicators, or because clinical trials have measured outcomes that are not important in the assessment of routine care. Indicators that cannot be mapped to outcomes in RCTs will also be identified as they may have little or no research evidence to support their use.


This protocol describes a novel method for comparing indicators with research evidence in clinical trials, and will help to determine if indicators associated with the management of asthma in children are supported by research evidence. This study will also generate a list of measurements used in RCTs of asthma management in children.

A limitation is that it will not examine other levels of evidence beyond level 1 (RCT) evidence. Some indicators may not be appropriate for use in RCTs and may appear in different study types, such as economic evaluations. Further, the protocol does not investigate if the RCTs support the use of indicators, and only looks for evidence of their use as a proxy indicator of utility. Some RCT outcomes may be important in early phases of the evaluation of an intervention, but will have limited application in routine practice. This study is also limited by searching only PubMed and assessing only English language RCTs.

However, it is important because it provides a systematic and objective way of finding candidate level 1 evidence for clinical indicators. It will assist in identifying candidate indicators, identifying research associated with current indicators, and identifying potential inconsistencies between research evidence and clinical practice.


View Abstract


  • Contributors MKC, GT and EC designed the study and drafted the manuscript. All the authors were involved in devising the protocol and read and approved the final manuscript.

  • Funding This work was supported by a National Health & Medical Research Council Program Grant APP1054146.

  • Competing interests None declared.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement This is a study protocol. Any data from the study can be requested from the corresponding author.

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.