Article Text

Investigating the cost-effectiveness of health information technologies: a systematic review protocol
  1. Aziz Sheikh1,2,
  2. Ulugbek B Nurmatov1,
  3. Kathrin Cresswell3,
  4. David Bates1,2
  1. 1eHealth Research Group, Centre for Population Health Sciences, The University of Edinburgh Medical School, Edinburgh, UK
  2. 2Division of General Internal Medicine and Primary Care, Brigham and Women's Hospital/Harvard Medical School, Boston MA, USA
  3. 3The School of Health in Social Science, The University of Edinburgh, Edinburgh, UK
  1. Correspondence to Professor Aziz Sheikh; aziz.sheikh{at}


Introduction There is a need to develop new, more cost-effective models of healthcare and in this vein there is a considerable international interest in exploiting the potential offered by major developments in health information technologies (HITs). Very substantial investments are, as a result, now being made globally, but these still probably only represent a fraction of the investments needed if healthcare is to make the transition from the paper to the digital era. Investing greater resources is, however, inherently challenging and unpopular at a time of financial austerity and this is furthermore complicated by the thus far variable evidence of health benefits and demonstrable short-term to medium-term returns associated with investments in HITs.

Objectives Building on our related systematic overviews investigating the impact of HITs, we now seek to estimate the cost-effectiveness of HITs and as a secondary aim to identify potentially transferable lessons in relation to how to realise returns on investments in these technologies.

Methods We will conduct a systematic review to identify the empirical evidence base surrounding the return on investments from implementing HITs. Two reviewers will independently search major international databases for published, unpublished and on-going experimental and quasi-experimental studies of interest published during the period 1990–2013. These searches of bibliographic databases will be supplemented by contacting an international panel of experts. There will be no restriction on the language of publication of studies. Studies will be critically appraised using the Critical Appraisal Skills Programme (CASP) Economic Evaluations checklist. In view of the anticipated heterogeneity in intervention investigated, study design and health system contexts, we will undertake a descriptive, narrative and interpretative synthesis of data.

Ethics and dissemination Ethical approval is not required.

Results These will be presented in one manuscript. The protocol is registered with the International Prospective Register for Systematic Reviews (PROSPERO) CRD42013005294.

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See:

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Strengths and limitations of this study

  • We will systematically identify and critically appraise the available evidence and grey literature on this important policy question and identify research in progress.

  • The main limitations are those that are common to all systematic reviews, namely the possibility of failing to identify all relevant evidence and the need to ensure that the synthesis of evidence is appropriate and meaningful.


Health systems globally face substantial challenges associated with, among other issues, rapidly changing demographic profiles (in particular, ageing populations and the reduced proportion of the population, ie, economically active), increasing numbers of people living with long-term conditions and associated spiralling healthcare costs, and the substantial, persistent variation in the quality and safety of care provision.1 ,2

There is, in response to these challenges, considerable policy interest in the potential of health information technologies (HITs) to support fundamental and far-reaching transformations of health systems, but achieving change on the scale required is, as our recent work and that of others has clearly demonstrated, challenging, time-consuming and expensive.3–6 Policy makers are therefore increasingly facing major challenges. These include questions surrounding the justification of substantial investments in HIT, particularly in times of austerity and unprecedented cuts in public investment, and associated pressures to achieve returns on investment in the short to medium terms.7 ,8

For example, as in many other parts of the world, the USA and the UK governments have already committed to making the transition from paper-based health systems to digital health economies.9 ,10 Substantial investments have thus been made through the Health Information Technology for Economic and Clinical Health (HITECH) Act in the USA and the National Programme for Information Technology (NPfIT) in the UK.11 These however represent first-wave investments in basic underpinning technologies and infrastructures associated with electronic health record (EHR) functionality. Very substantial subsequent investments will be needed in order to build on these infrastructures and utilise data held within them to redesign health systems.4 ,12

Given these challenges, there is a pressing need to better understand issues relating to the cost-effectiveness of HITs and also to identify strategies of proven value in enhancing economic returns on investments in HITs. This systematic review builds on our recent systematic overviews of the evidence investigating the impact of HITs,4 ,13 and is the first step in a related phased programme of work investigating approaches to enhancing returns on investments in HITs.

Methods and analysis


We will undertake a systematic review of the published, unpublished and in-progress empirical research literature.

Eligibility criteria

Interventions of interest

We are interested in a broad range of HIT, which include tools to support the management of data (eg, EHRs), support and enhance clinical decision- making (eg, computerised physician order entry and computerised decision support systems) and provide care at a distance (eg, telehealthcare).4 ,13 ,14

Study design

The following study designs will be potentially eligible for inclusion:

  • Systematic reviews, randomised controlled trials (RCTs), quasi-RCTs, controlled clinical trials, controlled before-after-studies, interrupted time series and before-after or cohort type evaluations, undertaken with formal health economic evaluations from the perspective of healthcare providers.

  • Economic modelling studies.

Outcome measures

  • Primary outcome

    • Estimates of the range of cost-effectiveness of HITs.

  • Secondary outcomes

    • Description of HITs that have been subjected to formal economic evaluation, the geographical, policy and clinical contexts in which these deployments have been made, the time frames over which the economic evaluations have been undertaken and the lessons learned.

    • Insights into interventions that are potentially cost-effective based on economic modelling studies.

Search methods

We will search the published empirical and grey literature from 1990 until 2013 for work investigating the returns on investment in HITs. This start date has been chosen to ensure that we capture the period when policy makers particularly became interested in the potential of HIT and also because the technologies that existed before this time period are likely to have only limited relevance to contemporary health systems.4 ,13 We will search the following major biomedical databases: The Cochrane Library (and its associated databases, namely: Cochrane Database of Systematic Reviews; Cochrane Central Register of Controlled Trials (CENTRAL); Cochrane Methodology Register; Database of Abstracts of Reviews of Effects (DARE); Health Technology Assessment Database; and the NHS Economic Evaluation Database), MEDLINE, EMBASE, Google Scholar, and Current Controlled Trials. Specific search strategies will be employed for each database; an example of a search strategy developed for MEDLINE is shown in online supplementary appendix 1.

Study selection

Titles and abstracts of studies identified from the searches will be checked by two investigators. The full text of all retrieved potentially eligible studies will be independently assessed against the above criteria by these two investigators. The investigators will decide which of the studies satisfy the inclusion criteria and the methodological quality of eligible studies will be recorded (discussed below). Any disagreements will be resolved by discussion between the investigators, with referral to a third member of the research team, if necessary.

Quality assessment and analysis

Formal quality assessment of eligible studies will be independently undertaken by two reviewers using the Critical Appraisal Skills Programme Economic Evaluations checklist.15 Disagreements will be resolved through discussion, with arbitration by a third reviewer if agreement cannot be reached through discussion.

Data extraction

Data will be abstracted onto a customised data extraction sheet by two independent reviewers; any disagreements will be resolved through discussion, with arbitration by a third reviewer if necessary. Variables to be extracted will include: author and year; title of the study; country of origin; healthcare setting; HIT application(s); estimates of costs and cost-effectiveness; time frame over which evaluations are undertaken; lessons learned. Key findings from each study will be summarised and presented in summary tables.

Data analysis

A quantitative synthesis is likely to prove inappropriate due to the heterogeneity of technologies, study designs and care contexts. Data will therefore be descriptively summarised and narratively synthesised. We will follow four main steps in conducting an interpretive synthesis of our findings: (1) noting the range of functions and uses of existing systems; (2) developing a preliminary synthesis of the findings of included studies; (3) exploring relationships in the findings and (4) exploring the potential transferability of findings to other contexts and settings.16 There are likely to be challenges associated with integrating heterogeneous data sources, due to for example, differences in interventions studied, methods, outcomes, study population and context.17 We will attempt to address these by judging the relevance of heterogeneity, and the quality of evidence and consistency of findings of included studies.17


The authors wish to record their appreciation to Andrew Stoddart, health economist at the Lothain Health Services Research Unit, University of Edinburgh for his advice.


Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Files in this Data Supplement:


  • Contributors AS and DB conceived the idea for this study. AS developed the methods, and together with UBN, KC and DB drafted this protocol.

  • Competing interests AS is supported by The Commonwealth Fund, a private independent foundation based in New York City. The views presented here are those of the author and not necessarily those of The Commonwealth Fund, its directors, officers, or staff.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement The systematic review protocol is registered with the PROSPERO International Prospective Register of Systematic Reviews (http:// CRD42013005294 and reported using Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA) guidelines19 in the peer-reviewed literature. The systematic review will be in one manuscript and shared with others.