Article Text

Download PDFPDF

Positioning whole exome sequencing in the diagnostic pathway for rare disease to optimise utility: a protocol for an observational cohort study and an economic evaluation
  1. Robin Z Hayeems1,2,
  2. Francois Bernier3,4,
  3. Kym M Boycott5,6,
  4. Taila Hartley5,
  5. Christine Michaels-Igbokwe4,7,
  6. Deborah A Marshall4,7
  1. 1Child Health Evaluative Sciences, Hospital for Sick Children Research Institute, Toronto, Ontario, Canada
  2. 2Institute of Health Policy Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
  3. 3Department of Medical Genetics, Alberta Children's Hospital, Calgary, Alberta, Canada
  4. 4Cummings School of Medicine, University of Calgary, Calgary, Alberta, Canada
  5. 5Department of Genetics, Children’s Hospital of Eastern Ontario Research Institute, Ottawa, Ontario, Canada
  6. 6Department of Paediatrics, Facuty of Medicine, University of Ottawa, Ottawa, Ontario, Canada
  7. 7O’Brien Institute for Public Health, University of Calgary, Calgary, Alberta, Canada
  1. Correspondence to Dr Robin Z Hayeems; robin.hayeems{at}


Introduction Despite the superior diagnostic performance of exome and genome sequencing compared with conventional genetic tests, evidence gaps related to clinical utility and cost effectiveness have limited their availability in routine clinical practice in many jurisdictions. To inform adoption and reimbursement policy, this protocol provides a chain of evidence approach to determining the diagnostic utility, clinical utility and cost-effectiveness of whole exome sequencing (WES) from seven medical genetic centres in two Canadian provinces.

Methods and analysis Using a multicentre observational cohort design, we will extract data specific to the pre-WES diagnostic pathway and 1-year post-WES medical management from electronic medical records for 650 patients with rare disease of suspected genetic aetiology who receive WES. The date from the clinical record will be linked to provincial administrative health database to capture healthcare resource use and estimate costs. Our analysis will: (1) define and describe diagnostic testing pathways that occur prior to WES among patients with rare disease, (2) determine the diagnostic utility of WES, characterised as the proportion of patients for whom causative DNA variants are identified, (3) determine the clinical utility of WES, characterised as a change in medical management triggered by WES results, (4) determine the pattern and cost of health service utilisation prior and 1 year following WES among patients who receive a diagnosis, do not receive a diagnosis, or receive an uncertain diagnosis and (5) estimate the cost-effectiveness of WES compared with conventional diagnostic testing pathways, measured by the incremental cost per additional patient diagnosed by WES using simulation modelling.

Ethics and dissemination This protocol was approved by Clinical Trials Ontario (CTO-1577) and research ethics boards at the University of Calgary (REB18-0744 and REB20-1449) and University of Alberta (Pro0009156). Findings will be disseminated through academic publications and policy reports.

  • Health policy

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.


  • This represents the largest Canadian clinical genetics cohort for which multiple dimensions of the value of whole exome sequencing (WES) (ie, diagnostic utility, clinical utility and cost-effectiveness) are assessed concurrently.

  • A real-world evidence and expert-informed framework was developed to support the economic evaluation described herein and represents a strength of this approach. However, this framework is limited by its specificity to a largely paediatric rare disease population and current practice patterns in Canada.

  • A non-WES comparator group could not be included in the study design. As such, counterfactuals required for modelling purposes rely on estimates from the literature and expert opinion.


The journey to diagnosis for a patient with a suspected rare disease can be long, expensive and often unsuccessful, adversely impacting patient care.1–4 Until recently, the identification of disease-causing DNA mutations in a person’s genome was a laborious process. The introduction of next-generation sequencing technologies has created a paradigm shift in our approach to the diagnosis of genetic disease. Hypothesis-free strategies, such as whole exome sequencing (WES) which interrogates the 2% of the genome that encodes proteins, and whole genome sequencing (WGS) which interrogates both coding and non-coding regions, have demonstrated significant diagnostic utility for rare disease.5 Compared with conventional genetic tests such as chromosome microarray and targeted gene panels, the diagnostic yield of WES and WGS have been reported to be 2–3 fold higher.6

Evidence gaps with respect to the clinical utility and cost-effectiveness of WES and WGS have limited their availability in routine clinical practice in many jurisdictions.7–9 Unlike prospective clinical research where the effectiveness of an intervention can be tied to a specific health outcome, the concept of clinical utility in genetic medicine is rarely uniformly defined nor directly tied to a specific health outcome. As such, generating and adjudicating evidence of clinical utility and cost-effectiveness is complex.10–13 Applying Fryback and Thornbury’s hierarchical model of efficacy to genomics,14 the constructs of utility and cost-effectiveness can be characterised as a chain of evidence, rather than placing emphasis on diagnostic yield alone.15 For example, genetic testing provides information that guides prognostication and medical management, which in turn can influence patient-related health and non-health outcomes. While a recent review of the utility of WES and WGS extended beyond diagnostic yield to include medical management outcomes, these outcomes were reported inconsistently and only half as often as diagnostic yield.16 With respect to estimating the economic impacts of sequencing, a recent systematic review identified 36 studies.17 Estimates of test costs ranged from US$55518 to US$516919 20 for WES and from US$190621 to US$24 81022 for WGS. Most studies concluded that WES and WGS were superior to other conventional testing methods in terms of incremental cost per additional diagnosis. While informative, many of these analyses were based on small samples sizes and provided limited detail on the components included in cost estimates. The authors of this review, as well as those who have completed more recent economic evaluations, conclude that knowledge gaps remain with respect to comprehensive measures of value and value for money of WES and WGS.23–26

To address these gaps, the protocol described herein reflects on a chain of evidence approach to determining the diagnostic utility, clinical utility and cost-effectiveness of WES at various points in the diagnostic journey for patients with rare disease in Canada.


The specific aims of this study are:

  1. To define and describe diagnostic testing pathways that occur prior to WES among patients with rare disease for whom a genetic aetiology is suspected.

  2. To determine the diagnostic utility of WES, characterised as the proportion of patients for whom causative variants are identified.

  3. To determine the clinical utility of WES, characterised as a change in medical management triggered by WES results.

  4. To determine the pattern and cost of health service utilisation from birth to 1 year following WES among patients who receive a diagnosis, do not receive a diagnosis, or receive an uncertain diagnosis.

  5. To estimate the cost-effectiveness of WES relative to conventional diagnostic testing pathways, as measured by the incremental cost per additional patient diagnosed via WES.


Design and settings

This is a multicentre observational cohort study of 650 patients who will receive clinical WES for the purpose of establishing a genetic diagnosis. For each patient enrolled between 2019 and 2022, diagnostic investigations performed prior to WES are collected retrospectively. One-year outcomes are collected prospectively (figure 1). The study settings include five genetics clinics in the Canadian province of Ontario and two genetics clinics in province of Alberta. Since four sites are children’s hospitals, the majority of participants are anticipated to be <18 years of age. Patients are referred to these clinics from specialists (eg, paediatricians, neurologists) or primary care providers.

Figure 1

Look-back window: earliest record of diagnostic investigation recorded in medical record. For administrative data, start date is date of birth.

Sample and recruitment

Eligibility criteria align with the most recent position statement from The Canadian College of Medical Geneticists on implementing clinical sequencing.27 Specifically, patients are eligible for WES and for this study if a baseline clinical genetics evaluation has been completed and a genetic aetiology for the phenotype is suspected. Clinical presentations that include >2 of: (1) moderate to severe developmental or functional impairment; (2) multisystem involvement; (3) progressive clinical course; (4) differential diagnosis that includes >2 conditions that would require evaluation by separate gene panel tests; (5) suspected severe genetic syndrome for which multiple family members are affected, or where parents are consanguineous. Patients are ineligible if their clinical presentation is limited to: (1) isolated mild intellectual disability or learning disabilities, (2) non-syndromic autism, (3) isolated neurobehavioural disabilities or (4) isolated neuropsychiatric conditions. Based on clinical volumes and the expected distribution of eligible phenotypes, we estimated that 500 Ontarians and 150 Albertans would provide sufficient sample to populate each of the phenotypic categories and be feasible to achieve within the recruitment period. This represents the largest clinical genetics cohort for which WES-related diagnostic utility, clinical utility and cost data has been ascertained in Canada.

Eligible patients or their family members are informed of the study by a clinical geneticist, subspecialist with expertise in genetics, or a genetic counsellor during or soon after their clinical consultation. All patients are informed that access to WES is not contingent on study participation. If interested in participating, patients provide informed consent for medical record review and for access to administrative health data in their respective province; Ontario’s Institute for Clinical and Evaluative Sciences (ICES) or Alberta Health Services (AHS).

Data collection

Medical records

Data entry staff at each site were trained to review records and identify relevant datapoints prior to data entry. At a given site, the same individuals complete pre-WES data entry, post-WES data entry and quarterly data auditing functions. Data from medical records specific to each patient’s pre-WES diagnostic pathway and post-WES medical management are entered into Genomics4RD, a centralised data repository for rare disease research.28 The data collection tool includes pre-WES and post-WES modules. The pre-WES variables include: (A) patient characteristics, (B) DNA-based diagnostic tests performed to date, (C) non-DNA based diagnostic investigations performed to date, (D) specialists involved in ongoing care and (E) a record of the diagnostic tests that the responsible clinician would have ordered in the absence of WES. To ascertain what this clinician would have ordered in the absence of WES, the clinician completes a checklist at the time WES is ordered and their checklist responses are entered into Genomics4RD (ie, hypothetical care pathway; table 1). We requested this information to inform our cost-effectiveness modelling analysis (aim 5) since it was not feasible to have a non-WES comparator groupincluded in the study design.

Table 1

Data collection—timeline, variables and data sources

Administrative health databases

Both Ontario and Alberta have single-payer, government-administered healthcare systems that provide services free at the point of access for residents with a valid provincial health card. All publicly funded services accessed are recorded within health administrative datasets. In Ontario, patient-level data and episodes of care will be linked across various administrative databases using the encrypted ICES key number. In Alberta, the analogous patient level linked data will be obtained using personal health numbers (PHNs), a unique, nine-digit PHN, which is used when accessing healthcare services (table 1).


Costs for laboratory tests (basic biochemistry, small-molecule disorders, mitochondrial diseases, peroxisomal diseases) will be based on the 2020 Schedule of Benefits for Laboratory Services from the Ontario Health Insurance Plan (OHIP). For diagnostic procedures (imaging, biopsies and invasive procedures, electrical activity studies), the primary costing sources will be the 2020 Schedule of Benefits, Physician Services OHIP and the Schedule of Facility Fees. Prices for genetic tests will be obtained from the laboratories that performed these tests, as recorded in Genomics4RD (ie, local or external laboratories). For administrative health datasets, costs associated with hospitalisations, emergency department visits, procedures, tests, physician visits and medications will be included, and costs associated with genetic counselling will be excluded. For cases in Ontario, validated standard ICES costing algorithms will be applied.29 For cases in Alberta, individual emergency department visits and hospitalisation costs will be estimated using the most recent resource intensity weights.30 The Alberta Ambulatory Care Classification System Interactive Health Data Application will be used to value emergency department visits and imaging. Unit costs for prescribed medications will be from Alberta Blue Cross Interactive Drug Benefit List. All costs will be reported in 2020 CAD.

Data analysis

Aim 1: diagnostic testing pathways prior to WES

Based on diagnostic investigations captured in the pre-WES period, patients will be assigned to groups that reflect the complexity of their diagnostic pathway. To organise these pathways, we engaged in an expert-driven consensus process.31 Using professional guidelines related to diagnostic algorithms for rare disease diagnosis as a reference point,32–34 we asked three medical geneticist coinvestigators to assist with developing a framework for categorising tests recorded in our dataset. Categorising tests as indicator and non-indicators tests, we established the SOLVE Framework for organising pre-WES diagnostic pathways.35 Specifically, indicator tests are defined as those with high specificity for diagnosing rare diseases and likely to contribute specific information towards achieving a clinically valid molecular diagnosis. Non-indicator tests are defined as those performed as a routine part of a diagnostic workup for a patient referred for evaluation of a rare disorder. Indicator tests are typically higher cost, potentially invasive, less accessible and ordered/interpreted by a subspecialist and non-indicator tests are typically lower cost, non-invasive, locally accessible and ordered/interpreted by a generalist.31 Guided by this framework, we will use frequency counts and descriptive statistics to summarise the number, type and cost of diagnostic tests per person in each type of diagnostic pathway and the time to diagnosis for each pathway. Diagnostic pathways will serve as comparator groups for the economic evaluation (aim 5).

Aim 2: diagnostic utility of WES

We will also determine the proportion of cases for whom WES identifies variants that are pathogenic, likely pathogenic or of uncertain clinical significance and the proportion of cases for which WES establishes a diagnosis, a partial diagnosis, a potential diagnosis and no diagnosis. The diagnostic yield of WES (ie, proportion of cases who receive diagnostic results) will be the primary measure of diagnostic utility, a core grouping variable for the clinical utility and health service utilisation analyses (aims 3, 4), and the primary outcome for the cost-effectiveness analyses (aim 5).

Aim 3: clinical utility of WES

We will summarise data related to the medical management implications of WES using descriptive statistics. We will determine the type and volume of management activities overall and per person. Where sample size permits, point estimates for change in medical management overall and for specific types of medical management will be compared statistically among those who receive a diagnosis, a potential diagnosis and no diagnosis and among pre-WES diagnostic pathway groups established in aim 1. We will examine the relationship between clinical characteristics and management change(s) using parametric or non-parametric statistics as appropriate. If indicated, we will construct regression models to determine predictors of changes in medical management.

Aim 4: pattern and cost of healthcare utilisation pre-WES and post-WES

Rates of outpatient visits, laboratory testing, imaging services, emergency department visits, admissions and associated costs will be compared among those who receive a diagnosis and those who do not. Utilisation rates and costs will be compared pre-WES and 1-year post-WES. We will use standard methods for comparing proportions, and Poisson regression to test whether there are significant differences in the volume, type and costs of service utilisation in the presence/absence of a diagnosis. We will examine total volume of activities based on the distribution of healthcare resource use as well as mean and median number of activities per patient pre-WES and post-WES. The total costs for each pre-WES and post-WES pathway will be summed and results will be grouped according to the WES result type.

Aim 5: cost-effectiveness of WES at different timepoints in the diagnostic pathway

The cost-effectiveness analysis will assess the incremental cost associated with WES at different points in the diagnostic pathway from the healthcare payer perspective, with diagnostic yield of WES as the primary measure of effectiveness. Change in medical management will be used as a secondary measure of effectiveness. To facilitate these comparative analyses, we will develop a simulation model to reflect alternative diagnostic pathways to achieve a molecular diagnosis, as defined in aims 1 and 2.36–39 The diagnostic pathway is marked by the number of events (i.e. indicator tests other than WES as defined by the SOLVE Framework).35 Each patient’s timeline is modelled according to the time between events and the resource utilisation associated with events. The time of each event is determined by the presence of an indicator test. Informed by our expert-driven consensus process,31 patients will be grouped according to the number of events observed (ie, 1, 2, 3 or 4+ events). The number of events and the resource utilisation and costs incurred for each time period will be informed by observed data; the model will draw probabilistically from the observed distributions for each subgroup. Ultimately, the model will simulate patients’ test trajectory, evaluating the cost-effectiveness of performing WES at distinct time points within the testing sequences. To populate the diagnostic trajectory for the non-WES comparator group, the estimates for the probability of diagnosis via WES at different timepoints will be informed by our expert-driven consensus process. Deterministic and probabilistic sensitivity analysis and expert-defined scenario analyses will be conducted to define model uncertainty and the incremental cost effectiveness ratios will be assessed using cost-effectiveness acceptability curves at multiple levels of willingness-to-pay.40 41

Patient or public involvement

Given the policy relevance of our work, this protocol was designed in collaboration with key decision-maker partners at The Ontario Ministry of Health and AHS. Our findings will inform recommendations related to the clinical implementation of WES for each Canadian province. Patients were not involved in the codesign of this work.

Ethics and dissemination

The research protocol for Care for Rare Solve was approved by Clinical Trials Ontario (CTO-1577), the provincial platform responsible for approving clinical trials and observational studies involving two or more academic or healthcare institutions in Ontario, and by research ethics boards at the University of Calgary (REB18-0744 and REB20-1449) and University of Alberta (Pro00091561). Findings will be disseminated through academic publications and policy reports.


We acknowledge several limitations. First, the distinction between indicator and non-indicator tests in the SOLVE Framework relies on expert opinion and is specific to a largely paediatric rare disease population in Canada and ordering practices that reflect currently available tests. Assessment of its face validity over time is warranted. Second, the availability of WES in Canadian clinics, through exceptional access programmes to US-based laboratories, precluded the inclusion of a non-WES comparator group. As such, counterfactuals required for modelling purposes will rely on estimates from the literature and expert opinion. Third, our sample size was informed by projected case volumes and was not hypothesis driven. Finally, a 1-year post-WES observation period presents only a short-term view of medical management impacts and resource utilisation patterns for this patient population. Longer-term impacts are not included. Limitations notwithstanding, this analysis is unique in its use of real-world data and is the largest cross-provincial analysis of the diagnostic utility, clinical utility, and cost effectiveness of clinical WES in a Canadian rare disease population. Findings may be used to inform the clinical genetics service delivery models in Canada and internationally.

Ethics statements

Patient consent for publication



  • Contributors RZH, DAM and CM-I contributed substantially to conception and design. RZH, FB, KMB, TH, CM-I and DAM drafted the article or revised it critically for important intellectual content. RZH, FB, KMB, TH, CM-I and DAM gave final approval of the version to be published.

  • Funding This work was supported by Genome Canada and the Ontario Genomics Institute (OGI-147), the Canadian Institutes of Health Research (GP1-155867), Ontario Research Fund, Genome Alberta, Genome British Columbia, Genome Quebec, and the Children’s Hospital of Eastern Ontario Foundation. Where not provided, award numbers are not available. Robin Z. Hayeems is supported by a Canada Research Chair in Genomics and Health Policy (Tier 2).

  • Competing interests None declared.

  • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

  • Provenance and peer review Not commissioned; externally peer reviewed.