Article Text
PDF
Abstract
Introduction Recent avoidable failures in patient care highlight the ongoing need for evidence to support improvements in patient safety. According to the most recent reviews, there is a dearth of economic evidence related to patient safety. These reviews characterise an evidence gap in terms of the scope and quality of evidence available to support resource allocation decisions. This protocol is designed to update and improve on the reviews previously conducted to determine the extent of methodological progress in economic analyses in patient safety.
Methods and analysis A broad search strategy with two core themes for original research (excluding opinion pieces and systematic reviews) in ‘patient safety’ and ‘economic analyses’ has been developed. Medline, Econlit and National Health Service Economic Evaluation Database bibliographic databases will be searched from January 2007 using a combination of medical subject headings terms and research-derived search terms (see table 1). The method is informed by previous reviews on this topic, published in 2012. Screening, risk of bias assessment (using the Cochrane collaboration tool) and economic evaluation quality assessment (using the Drummond checklist) will be conducted by two independent reviewers, with arbitration by a third reviewer as needed. Studies with a low risk of bias will be assessed using the Drummond checklist. High-quality economic evaluations are those that score >20/35. A qualitative synthesis of evidence will be performed using a data collection tool to capture the study design(s) employed, population(s), setting(s), disease area(s), intervention(s) and outcome(s) studied. Methodological quality scores will be compared with previous reviews where possible. Effect size(s) and estimate uncertainty will be captured and used in a quantitative synthesis of high-quality evidence, where possible.
Ethics and dissemination Formal ethical approval is not required as primary data will not be collected. The results will be disseminated through a peer-reviewed publication, presentations and social media.
Trial registration number CRD42017057853.
- health economics
- health policy
- quality in health care
- patient safety
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
A strength of the proposed study is the application of transparent and replicable procedures for searching health economic literature.
The breadth of the search strategy is a strength as it lends to generating findings that are translational.
A weakness of this study is the close but imperfect replication of previous reviews. This limits the comparability of this study’s findings with others’, which also limits our interpretation of the maturation of evidence.
Considering previous findings, a large sample size for statistical pooling of effect size(s) within subgroups is not anticipated. This may limit the strength of this protocol as a means of producing a quantitative synthesis.
Introduction
After recent focusing events in which avoidable failures in healthcare were identified, patient safety remains high on the clinical and policy agenda in health systems. The burden of lapses in safety is high in developed health systems. In the USA, the Office of the Inspector General (OIG) estimated an adverse event (AE) rate of 13.5% for hospitalised Medicare beneficiaries, amounting to approximately 180 000 additional deaths per annum.1 For the UK, the rate is estimated to be 10.8%, although there are limitations to this calculation and the methodology differs to that used by the US OIG.2 3 According to the most recent systematic reviews of economic analyses in patient safety, there is a dearth of high-quality evidence on the economic impact of avoidable harm and the interventions that aim to improve safety. As such, there is limited evidence to support resource allocation decisions in this area. Meltzer summarised the state of evidence in a 2012 editorial in which economic evaluation in patient safety was described as a ‘neglected necessity’.4
In light of findings by Etchells et al 5 and De Rezende et al,6 there is a need to conduct a new review of economic evaluations to measure recent progress in this area. A central question is whether the research community has produced more high-quality evidence since these publications. Implicitly, this question should be approached by (1) conducting a before and after comparison of studies using the same, valid quality assessment instrument and (2) this is a prerequisite for a potential metasynthesis of recent (last 10 years) evidence.
Etchells et al 5 restricted their search to comparative evaluations of 15 patient safety target conditions and 6 improvement strategies using a Medline and Agency for Healthcare Research & Quality (AHRQ) Patient Safety Network search between 2000 and 2011. Studies that did not analyse an intervention with a patient safety target or were not in the acute setting were excluded. Studies were also excluded if evidence of effectiveness was poor, according to the Cochrane collaboration risk of bias tool,7 or not cited. Those studies that were included were quality assessed using the Drummond checklist,8 which was completed for each study by two independent reviewers. Consensus was achieved based on a five-point discrepancy between assessments scores. The findings from this review show the paucity of high-quality comparative research.
From this review, only five studies were considered high quality: pharmacist-led medication reconciliation dominated a strategy of no reconciliation9; additional erythropoietin to reduce transfusion-related AEs were evaluated against the standard of care in another study10; use of chlorohexidine gluconate to reduce catheter-related infections was found to dominate povidone-iodine use11; the Keystone intensive care unit patient safety programme was evaluated using effectiveness data related to the reduction in central line-associated bloodstream infections and was found to dominate standard of care12; lastly, strategies to reduce retained surgical foreign bodies were compared and standard counting was found most likely to dominate a no counting strategy.13
De Rezende et al 6 used Medline, National Health Service (NHS) Economic Evaluation Database (EED), and Econlit databases to search for English and French language economic evaluations from 2000 to 2010. Their focus was on characterising the types of AEs studied and methods used to estimate costs and cost-effectiveness. Two independent reviewers used a common template to assess AE type, evaluation method, perspective, population studied, cost components, data sources, cost calculation methods, outcome measures and results obtained. As such, burden studies, cost-effectiveness studies and intervention cost analyses were identified. A quality assessment tool was conceived and used to determine studies’ methodological quality and discrepancies were discussed by the four authors.
Using a qualitative synthesis, this review found that most studies considered healthcare-associated infections and/or infection control activities and medication-related AEs. Three economic burden/cost of-illness studies, five complete economic evaluations and two intervention cost studies were described in detail by the research group because they were considered high quality (a published instrument was not applied for quality assessment). Economic evaluations of hospital deep cleaning programmes, staff vaccination against nosocomial pertussis, three medication-related AE reduction strategies, a national methicillin-resistant Staphylococcus aureus (MRSA) reduction policy and the introduction of electronic medical records to reduce prescription errors were identified. The findings from this review provide insight on the scope of economic analyses performed in this area.
Taken together, Etchells et al 5 and De Rezende et al 6characterise the literature from 2000 to 2011 with a combination of scope and quality appraisal. There is a renewed evidence gap in economic evaluations of patient safety for three reasons: (1) the reviews by these research groups are no longer recent; (2) the breadth of safety priorities and interventions and the depth of objective quality assessment in these reviews have historical and scientific limitations and (3) considerable resources continue to be invested in patient safety without a renewed synthesis of evidence to inform decision-making.
The proposed review will address these three concerns by expanding the scope of the review conducted by Etchells et al 5 and raising the scientific rigour compared with the De Rezende et al 6 review. This approach will be essential to determine the extent to which methodological progress has been made since Meltzer’s (2012)4 editorial. To ensure that the quality assessment is valid and reliable, high-quality studies published after 2007 that were also identified in previous reviews will be included for quality assessment in this study.
Research aims and objectives
This study aims to (1) assess methodological progress in economic studies in patient safety and (2) provide a quantitative summary and synthesis of the most recent evidence in this area.
Characterise and describe the types of economic analyses performed in patient safety since 2011;
Assess the quality of empirical (descriptive and comparative/experimental) economic analyses in patient safety;
Compare the scope and quality of economic evidence in patient safety between 2007 and 2011 to the end date of the literature search.
Methods and analysis
The scope of this study is driven by the combined aspects of the Etchells et al 5 and De Rezende et al 6 studies. The approach is derived using resources that enhance methodological transparency and improve the systematic nature of the search strategy and evidence synthesis. These resources are described for both the economic and medical aspects of this review. Medline, Econlit and NHS EED bibliographic databases will be searched from January 2007 using a combination of medical subject headings (MeSH) terms and research-derived search terms. This approach is considered efficient for searching health economic literature.14 The primary outcomes of interest in this review are economic endpoints in patient safety research.
The overarching approach to this study is summarised in the prospective Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) (see online supplementary 1). A completed PRISMA Protocols checklist can be found in online supplementary 2. The following research stages will be followed:
Search
Title and abstract searches using MeSH and researcher-derived search terms (provided in table 1);
Combine database searches, deduplication using a prespecified procedure.
Search themes (facets) and terms derived for each theme relating to economic analyses in patient safety
Screening
Title and abstract screening using prespecified inclusion and exclusion criteria;
Full-text and reference list screening according to pre-specified inclusion and exclusion criteria.
Quality assessment
Risk of bias assessment and Drummond checklist assessment of economic evaluation quality;
Data extraction in accordance with research objectives.
Data synthesis
Evidence synthesis of scope and quality of evidence;
Presentation and publication of findings;
Assessment of evidence for use in subgroup meta-analyses.
Eligibility criteria
The systematic review does not focus on a particular population, disease area or health system setting as it intends to characterise the scope of all economic evaluations in patient safety. The search criteria are shaped around two core themes for original research in ‘patient safety’ and ‘economic analyses’. We define patient safety as ‘The avoidance, prevention and amelioration of adverse outcomes or injuries stemming from the process of healthcare’.15 Economic analyses in this study broadly refer to cost-of-illness, cost-effectiveness, cost utility, cost benefit, cost consequence, cost minimisation and cost analysis studies.
Search terms will be restricted to title and abstract only at the screening stage. The time frame for study selection is decided by the completion dates of previous reviews,therefore all studies published since January 2007. This also allows for overlap between the current study and previous reviews, ensuring appropriate articles are included.
Inclusion criteria
Participants/population: individuals of any age and gender, all index admissions and in all health system settings who are at risk of experiencing a safety-related AEs;
Intervention(s): this review is deliberately broad and is designed to include descriptive and intervention studies. Interventions refer to any care process that intends to reduce the incidence of safety-related AEs. These may include new drugs (or drug-presentations), procedures, guidelines and policies that are produced to improve patient safety;
Comparators: all comparison groups;
Outcomes: all economic and clinical outcomes related to patient safety will be considered.
Exclusion criteria
Participants/population: articles that report populations not at risk of iatrogenic harm;
Intervention(s): any studies reporting on non-safety-related interventions that is, those that do not explicitly intend to reduce the rates of harm;
Comparators: no specific exclusion;
Outcomes: studies with outcomes that are not considered a consequence of patient safety interventions will be disregarded;
Timing: studies published during or before December 2006;
Studies that do not report economic data.
Other general exclusion criteria
Non-English language articles;
Opinion: letters, editorials and commentaries;
Systematic reviews and book chapters;
Partial or non-peer reviewed evidence: conference abstracts, study protocols.
The review will consider both experimental and epidemiological study designs that are incorporated with economic analyses. We will therefore include randomised controlled trials, non-randomised controlled trials, quasi-experimental before and after studies, prospective and retrospective cohort studies, case–control studies and analytical cross-sectional studies. Only peer-reviewed publications of original research will be considered, and these criteria are satisfied by applying the Neyt and Chalon optimised procedure for searching economic evaluations in Medline.16 This procedure may be used exclusively or in combination with the research group’s search terms.
Screening
Initial search result titles, abstracts and index or keyword terms will be screened. Studies will be screened by two reviewers acting independently. Reference lists of all identified reports and articles will be searched for other relevant studies.
After title and abstract screening, full-text reviews will be conducted.
Exclusion criterion
Studies that do not investigate economic outcomes;
Studies that do not investigate patient safety.
The two reviewers will extract relevant data on study design, funding, population(s), setting(s), disease area(s), intervention(s), outcome(s) and effect size(s). These data will be used to for the quality assessment of the review.
Quality assessment
Two reviewers will independently assess the risk of bias in studies using the Cochrane collaboration risk of bias tool.7 This tool will be applied to assess the quality of effectiveness data reported and used for cost-effectiveness analyses. It will also be used to assess epidemiological and observational findings. Overall, the quality assessment will consider the following study-level aspects:
Randomisation sequence generation: was the allocation sequence generated appropriately?
Treatment allocation concealment: were study participants, clinicians and other healthcare or research staff unaware of the intervention(s) given at the enrolment stage?
Blinding: were the personnel assessing outcomes and analysing data blinded to the intervention allocated during the trial?
Completeness of outcome data: were participant exclusions, attrition and incomplete outcome data adequately reported in the published manuscript?
Selective outcome reporting: is there evidence of selective outcome reporting and could this have biased the study results?
Other sources of bias: was the study apparently free of any other high-risk sources of bias?
Disagreements between the review authors over the risk of bias studies will be resolved by discussion, with arbitration by a third reviewer where necessary. For studies considered to have a low risk of bias and that also report economic evidence, the Drummond checklist for assessing the methodology quality of economic evaluations will be applied. Two independent reviewers will score these (out of 35). Scores will be reviewed by a second reviewer and any disagreements in score of >5 will be resolved with arbitration by a third reviewer.
Data synthesis
A descriptive (qualitative) analysis of the scope of economic evidence in patient safety will be presented. Consistent with De Rezende et al,6 the types of economic analyses will be presented as studies of either economic burden/cost of illness or cost analysis of interventions or full economic evaluations (encompassing cost and effectiveness data).
The populations (by demographic), settings (primary, secondary, community or social care), diseases areas (problems of circulation, cancers and tumours, mental disorders, musculoskeletal system and connective tissue) and interventions (safety-related AE that is targeted—either infection-related, medication AE, surgical safety, hospital accidents) of all economic analyses will be presented.
A synthesis of quality scores for risk of bias and economic evaluation quality will be given. Effect size(s) and cost-effectiveness results of high-quality papers—those scoring above 20 on the Drummond checklist—will be reported. This cut-off score is consistent with that used by Etchells et al.5 Sensitivity analysis results will also be given, if reported. The sensitivity of review findings to stratification of evidence quality will also be explored.
A quantitative synthesis will be performed if necessary data are available. High-quality findings from comparative analyses will be compared with findings from Etchells et al 5 and De Rezende et al’s6 reviews. High-quality findings identified in these reviews and that were published after 2006 will be included for quantitative synthesis. Effectiveness estimates will be grouped by the type of safety-related AE that are the target of interventions identified in the review (for example, nosocomial infections). Estimates will then be pooled for statistical meta-analysis. Heterogeneity will be assessed statistically using an appropriate test statistic (such as χ2). Effect sizes expressed as OR (for categorical data) and weighted mean differences (for continuous data) and their 95% CIs will be calculated for analysis. Where statistical pooling is not possible, findings will be tabulated for comparison.
Metabiases will be determined using the risk of bias and Drummond checklist scores. These results will be explored for high, moderate, low and very low-quality evidence. Systematic biases associated with the quality of evidence will be presented.
Amendments
Amendments to this protocol will be captured and documented with reference to saved searches and analysis methods, which will be recorded in bibliographic databases (OVID), Endnote and Excel templates for data collection and synthesis. Deviations to this protocol will be documented in a research manuscript that will be submitted for peer-reviewed publication.
Discussion
In light of the findings in previous reviews of economic analyses in patient safety, this review is a timely update on a high-priority topic in healthcare. A strength of the proposed study is the application of transparent procedures for searching health economic literature. These procedures are (1) application of the Neyt and Chalonprocedure for economic literature searches in MEDLINE, which is a published procedure,16 (2) full publication of both MeSH and research-group derived search terms associated with each theme/facet, (3) use of a reproducible deduplication procedure for Endnote article management software. Second, we aim to address two aspects—scope and quality—to determine how research in this area has matured since 2011. This is important because many health systems are facing efficiency and sustainability challenges, which require robust economic evidence to support decision-making in patient safety. This review aims to support targeted decision-making in the medical profession that is, synthesising the evidence by population, setting and disease area. The breadth of the search strategy should therefore improve the translational aspects of the anticipated research findings. A weakness of this study is the close, but imperfect replication of previous reviews. This limits the comparability of this study’s findings with others’, which also limits our interpretation of the maturation of evidence. In addition, considering previous findings, a large sample size for statistical pooling of effect size(s) within subgroups is not anticipated. This may limit the strength of this protocol as a means of producing a quantitative synthesis.
Acknowledgments
The composition of the search terms and procedural aspects of the search strategy were significantly improved with support from Jacqueline Cousins, information specialist at the Imperial College London.
References
Footnotes
Contributors AC conceptualised this research, designed and wrote the protocol. RM contributed to the conceptualisation and commented on multiple versions of the protocol. EMr contributed to the conceptualisation and commented on multiple versions of the protocol. JM contributed to the conceptualisation and commented on multiple versions of the protocol. EM leads the economics theme of the NIHR PSTRC at Imperial College and contributed to the conceptualisation of this work and commented on the penultimate draft. AD provides broader oversight of the NIHR PSTRC at Imperial College and contributed to the conceptualisation of this work and commented on the penultimate draft.
Funding This work is supported by the National Institute for Health Research (NIHR) Patient Safety Translation Research Centre. The views of the authors do not necessarily reflect those of the NHS, NIHR or the Department of Health.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.