Article Text
PDF
Abstract
Introduction Clinical practice guidelines facilitate optimal clinical practice. Point of care access, interpretation and application of such guidelines, however, is inconsistent. Informatics-based tools may help clinicians apply guidelines more consistently. We have developed a novel clinical decision support tool that presents guideline-relevant information and actionable items to clinicians at the point of care. We aim to test whether this tool improves the management of hyperlipidaemia, atrial fibrillation and heart failure by primary care clinicians.
Methods/analysis Clinician care teams were cluster randomised to receive access to the clinical decision support tool or passive access to institutional guidelines on 16 May 2016. The trial began on 1 June 2016 when access to the tool was granted to the intervention clinicians. The trial will be run for 6 months to ensure a sufficient number of patient encounters to achieve 80% power to detect a twofold increase in the primary outcome at the 0.05 level of significance. The primary outcome measure will be the percentage of guideline-based recommendations acted on by clinicians for hyperlipidaemia, atrial fibrillation and heart failure. We hypothesise care teams with access to the clinical decision support tool will act on recommendations at a higher rate than care teams in the standard of care arm.
Ethics and dissemination The Mayo Clinic Institutional Review Board approved all study procedures. Informed consent was obtained from clinicians. A waiver of informed consent and of Health Insurance Portability and Accountability Act (HIPAA) authorisation for patients managed by clinicians in the study was granted. In addition to publication, results will be disseminated via meetings and newsletters.
Trial registration number NCT02742545.
- health informatics
- adult cardiology
- internal medicine
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
The main strength of this study is the design of the clinical decision support tool. The tool was developed with iterative clinician input to fit the workflow. And it uses complex data in its algorithms (such as risk scores and information only available through natural language processing) to enable the provision of individualised treatment recommendations for patients with three distinct yet common cardiovascular conditions. It efficacy will be evaluated with a rigorous study design using clinically meaningful endpoints.
The main limitations are that the study may lack generalisability—as the clinician decision support tool is a proprietary system, not yet widely available, tested in an academic medical centre.
Background
Cardiovascular disease is the leading cause of morbidity and mortality in the USA.1 Prevention and treatment guidelines for cardiovascular diseases aim to improve outcomes and cost effectiveness.2–4 However, successfully incorporating these guidelines into clinical practice remains a challenge.
Some of the largest gaps between evidence-based guidelines and clinical practice exist for the management of hyperlipidaemia, atrial fibrillation and heart failure.5 It is estimated that only 41% of the 61.8 million patients in the USA who are eligible for statins under the 2013 American College of Cardiology/American Heart Association (ACC/AHA) guidelines are receiving these medications.6 Of the 2.8 million patients with heart failure and left ventricular ejection fraction (LVEF) <40%, rates of compliance with the American College of Cardiology Foundation (ACCF)-AHA guidelines are as low as 7.3% for some measures.7 Similarly for atrial fibrillation, guideline compliance is suboptimal.8 9
Many interventions have been designed to address these gaps between evidence and practice, yet most achieve only minimal gains in guideline adherence.10 Although informatics-based clinical decision support tools have shown promise, most interventions aimed at improving the medical management of cardiovascular disease have shown only minimal gains in compliance to recommended treatment, with a recent systematic review showing a median improvement of 2.0% across interventions.11 Among studies looking at interventions aimed at improving prescribing practices for hyperlipidaemia, atrial fibrillation and heart failure, results ranged from no effect to 13.3% improvement.12–15 The most successful intervention was unique in that the tool provided clinicians with all the information needed to make a decision within the application. None of the algorithms used by the tools were particularly sophisticated in their ability to take into account common clinical conditions that might alter treatment recommendations. Only two of the four interventions were integrated into the electronic medical record (EMR).
Using the lessons learned from these studies and also from other studies of clinical decision support for chronic disease management,16–18 we devised a tool that took into account individual patient factors that might alter treatment recommendations and presented these factors along with calculated risk scores, decision aids and educational materials when a treatment recommendation was made. Clinician input was sought throughout the design process and when deciding on how to integrate the tool into the clinician workflow.
In this paper, we first describe the development of this clinical decision support tool, MayoExpertAdvisor and then outline the protocol for an ongoing clinical trial to formally evaluate its effectiveness. We hypothesise that MayoExpertAdvisor will increase clinician adherence to best practice treatment recommendations for patients with hyperlipidaemia, atrial fibrillation and heart failure.
Development and validation of MayoExpertAdvisor
Like many institutions, Mayo Clinic develops and implements institution-wide guidelines to standardise patient care. Mayo Clinic’s care process guidelines incorporate expert opinion, best practices supported by professional organisations, and local standards of care while allowing for individualisation of treatment. These guidelines are available to Mayo Clinic clinicians through a multifaceted computer-based tool called AskMayoExpert. AskMayoExpert is available via local Intranet to all clinicians at Mayo Clinic and its affiliated healthcare organisations. While the knowledge contained in such care process guidelines is a useful reference for clinicians, a fundamental challenge is faced when clinicians must look up data within the EMR and/or calculate risk scores to apply care algorithms.19 20 As an adjunct tool to complement AskMayoExpert, MayoExpertAdvisor was developed to automate these tasks. MayoExpertAdvisor pulls data from the EMR and calculates risk to determine appropriate care. It then notifies the clinician when care differs from the guidelines by delivering action-oriented recommendations, along with a brief justification, in real time at the point of care. MayoExpertAdvisor is embedded in the EMR to facilitate integration into the clinical workflow.
Institutional guidelines for hyperlipidaemia, atrial fibrillation and heart failure
Using the process described above, institutional guidelines for management of hyperlipidaemia, atrial fibrillation and heart failure were developed. Publicly available resources used as a basis for the guidelines are listed in table 1. Generally, Mayo’s institutional guidelines mirror nationally recognised best practices. For example, the institutional guideline for hyperlipidaemia uses the 2013 ACC/AHA guidelines on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk as a starting point but also incorporates other factors, such as the patients Framingham 30-year risk of atherosclerotic cardiovascular disease (ASCVD) and the use of a decision aid to discuss the pros and cons of statin therapy with a patient.21 22 The institutional guidelines used to develop MayoExpertAdvisor for hyperlipidaemia, atrial fibrillation and heart failure are included in the online supplementary appendices 1–3.
Resources used for the development of institutional guidelines
MayoExpertAdvisor development
Operationally, MayoExpertAdvisor (MEA) captures the patient-specific data required to determine appropriate care recommendations from the EMR and then presents relevant data alongside treatment suggestions to clinicians. To develop MayoExpertAdvisor, a team of clinicians and information technology (IT) experts worked together to translate the institutional guidelines into computational logic. Then, the IT team developed processes to automatically gather from the EMR the data needed to run the algorithms. For data not optimally captured from discrete sources (such as the International Classification of Diseases Ninth Revision (ICD-9) or 10 codes), natural language processing (NLP) was used. For example, common comorbidities were captured both from ICD-9 or 10 codes and from free text within problem lists. Some inputs such as alcohol use to determine bleeding risk were almost exclusively captured through NLP as this is not routinely coded in a discrete format by clinicians.
Clinicians worked with user experience designers to develop the user interface. A user-centred design process in which the team researched the needs of clinicians and aligned the design of the application to meet those needs was used. This process started with collaborative codesign working sessions, in which information needs and decision-making flows were elicited from clinicians and user interface design concepts were created. Industry best practice design principles for usability23 were used throughout the design process. Once an initial design was agreed on, the team conducted task-based usability testing to uncover any issues users might have in completing key tasks with the design. The user interface design was then revised based on the findings from this usability study to better meet user needs.
Validation of MEA
Level 1: receiving the correct data
MEA requires complex data sources, such as risk scores and information only retrievable through NLP. The accuracy of each data element needed to run MEA algorithms was tested until the error rate was less than 5% in 30 consecutive patient records.
Level 2: testing of the logic
MEA logic was tested on patients with hyperlipidaemia, atrial fibrillation and heart failure and the automated recommendations were compared with clinician chart review. The IT team was notified when any discrepancies were found so that programming errors could be resolved. This process was repeated until no errors were observed in 200 patient records.
Level 3: testing of user interface
A formal usability study was conducted, in which 12 clinicians walked through clinical scenarios using a working MEA prototype. Clinicians were observed by a facilitator who recorded any difficulties noted and also asked clinicians a set of standardised questions at the end of each scenario and solicited general feedback about the tool. Based on study findings, the user interface was redesigned to address observed difficulties navigating the tool and clinician concerns. For example, there was a general lack of clarity among participants of where to look in the interface for which information. To address this issue, the information was reorganised around conditions rather than by type of information (eg, care recommendations, risk scoring tools and knowledge resources) and the visual hierarchy of information was adjusted by changing font treatments and adding graphical elements such as icons to direct the user’s vision. Another issue was users needing to see current therapies being used for the patient, but not having this information available in the tool. This was addressed by adding this information into the interface for each condition.
Level 4: pilot
MEA was tested in a clinical setting with 26 primary care physicians for 4 weeks. User feedback was informally solicited throughout the pilot and formally collected through a postpilot survey. Survey results indicated clinicians were generally satisfied with the tool, but thought care recommendation notification process, though email, did not fit their workflow. They also indicated they often did not follow a care recommendation due to a comorbidity not captured by the tool, with the most often cited comorbidity being statin intolerance. Additional capabilities for recognising comorbidities were built into the tool after the pilot (ie, identifying statin intolerance), and the tool was further integrated into the EMR through the additions of coloured MEA alerts visible within the EMR that notified the clinician of the presence of a MEA recommendation.
MEA for hyperlipidaemia, atrial fibrillation and heart failure
Clinicians are notified in several locations in the EMR when a recommendation is available for a patient (see figure 1). When a clinician clicks on an alert or the MEA tab in the EMR, a screen opens that reveals the treatment recommendation along with relevant patient data, risk scores, decision aids and educational materials (see figure 2). Clinicians can manipulate patient data to visualise how lifestyle modification (eg, smoking cessation) or treatment changes (eg, starting a drug or changing drug dose) will affect risk. These manipulations are only temporary and are not stored in MEA or in the EMR.
MEA alerts. The EMR notifies the clinicians of MEA recommendations in three ways. (1) In the patient appointment section, MEA is seen with the number of recommendations in parentheses next to the patient’s name. (2) MEA also appears as a clinical alert. (3) Within the EMR banner, there is also a MEA alert that opens the MEA screen. EMR, electronic medical record; MEA, MayoExpertAdvisor.
MEA user interface. (A) Care recommendation. Depending on the individual patient’s date in the EMR, MEA makes a recommendation. (B) Vitals. Most recent outpatient vital signs. (C) Relevant patient data. The most relevant demographics, conditions and lab results for managing the given condition. (D) Resources for next steps. Additional condition-specific tools (eg, list of moderate and high-intensity statins) to assist in recommendations. (E) Risk calculators. Condition-specific risk calculators with a patient’s data prefilled for real-time calculations. (F) Decision aids. Mayo-vetted shared decision-making tools. Field values are prefilled with patient data. ACC ASCVD, American College of Cardiology Atherosclerotic Cardiovascular Disease; EMR, electronic medical record; LDL, low-density lipoprotein; MEA, MayoExpertAdvisor.
There are several treatment recommendations that MEA can give for each cardiac condition. For hyperlipidaemia, MEA suggests a specific statin intensity. For atrial fibrillation, MEA decides in favour or against anticoagulation based on the CHA2DS2-VASc24 and the HAS-BLED scores.25 For heart failure, MEA recommends (1) initiation of beta blocker therapy, (2) uptitration of beta blocker therapy, (3) initiation of ACE inhibitor/angiotensin receptor blocker (ARB) therapy (4) and uptitration of ACE inhibitor/ARB therapy. MEA also notes when patients with the above conditions are well managed according to institutional guidelines, but this does not generate an alert. For a detailed list of possible MEA recommendations, please see table 2.
MEA recommendations
Methods
Study design
MEA is undergoing evaluation in an ongoing cluster-randomised, non-blinded clinical trial with clinician care teams as the unit of randomisation. If MEA is effective at the end of the trial, it will be made available to both the intervention and control arm.
Setting
The study is being conducted at four geographically distinct primary care practice sites affiliated with the Mayo Clinic in Rochester, Minnesota, an academic medical centre.
Eligibility criteria
All primary care clinicians (physicians, nurse practitioners and physician assistants) practising in the divisions of Family Medicine or Primary Care Internal Medicine at Mayo Clinic in Rochester, Minnesota were eligible to participate. Physicians in training (ie, medical students and residents) were excluded. Patient data were used for determination of study outcomes. To be included for data extraction purposes, patients needed to have one of the specified clinical conditions and to be in the panel of one of the clinicians in the trial. All patients seen by eligible clinicians are Olmsted County residents and thus the demographics reflect the local population. Patients who did not authorise the use of their medical record for research were excluded.
Recruitment of clinicians
Clinicians were recruited through presentations at division meetings, emails, phone calls or by individual face-to-face contact. If they indicated they were interested in participating, a meeting with a study team member was scheduled to review the details of the study. Any individual who indicated they did not wish to participate in the study was placed on a do not contact list and no further contact was initiated. To avoid potential coercion, clinicians were not recruited by anyone in a supervisory or position of perceived power over them.
Randomisation
Clinicians were cluster randomised by care teams to minimise the potential contamination that could occur if multiple members of a care team saw the same patient during the study period. Care teams varied in size from five to eight clinicians. There were 20 care teams representing three practice types (family medicine, internal medicine and combined family medicine and internal medicine), located at four distinct clinical sites. Prior to randomisation, these 20 teams were stratified into six strata to create balance of practice type and physical location. There was one mixed strata with three family medicine and one internal medicine care team pooled from two physical locations; the remaining five strata were homogeneous in physical location and practice type. Each stratum comprised two, four or six care teams. A stratified, blocked randomisation schedule was generated by the study biostatistician (REC) to produce an equal number of care teams per condition using a randomly generated code. The randomised assignments remained with the study biostatistician until the study start, at which time access to MEA was granted, if applicable, and consented clinicians were notified by study staff of the group assignment.
Clinician education (both groups)
All clinicians received a 4 min online educational module that reviewed the current institutional guidelines for management of hyperlipidaemia, atrial fibrillation and heart failure.
Intervention group
Clinicians in care teams assigned to the intervention arm received training by the research team on the use of MEA were given access to MEA and will be alerted of MEA care recommendations throughout the study period.
Control group
Clinicians in care teams assigned to the standard of care arm continue to have access to institutional guidelines through the AskMayoExpert knowledge resource but do not have access to the MEA screen. MEA recommendations are run in the background for data collection purposes (see ’Date sources' section below), but control group clinicians do not have access to these recommendations.
Data sources
Baseline assessment
The MEA algorithm was run on all eligible patient records at the start of the study. The specific nature of the MEA recommendations for each patient (if any) were recorded; if patients were already ‘well managed’ this was also recorded. These data will be used to assess the baseline adherence to the institutional guidelines for hyperlipidaemia, atrial fibrillation and/or heart failure.
Daily previsit MEA assessment of patient records
The MEA algorithm is applied to all eligible patients the evening prior to a scheduled outpatient visit. As with the baseline assessment, the daily previsit assessment includes the specific nature of the MEA recommendations for each patient and also which patients are well managed. These data will be used to assess the previsit adherence to the institutional guidelines for hyperlipidaemia, atrial fibrillation and/or heart failure.
Daily postvisit MEA assessment of patient record
The MEA algorithm is applied the evening after the visit to the same patients to whom the MEA algorithm was applied the evening prior and again extracts information about MEA recommendations for each patient. These data will be used to assess the postvisit adherence to the institutional guidelines for hyperlipidaemia, atrial fibrillation and/or heart failure.
Study completion MEA assessment
The MEA algorithm will be run on all eligible patients at the completion of the study to provide a similar set of data as described for the Baseline MayoExpertAdvisor Assessment.
MEA application measures of use
Clicks to view MEA recommendations are being tracked by standard timing and click metrics and are stored as a component of the application’s operation.
Survey data
A nine-item survey was sent to all study clinicians at the start of the study ascertaining (1) efficiency in managing the three clinical conditions; (2) frequency of using risk scores in managing the three clinical conditions and (3) the overall use/burden of clinical decision support tools. The same survey will be administered at 2-month intervals throughout the study. At the completion of the study, we will ask clinicians in the intervention arm to complete a 25-item survey to evaluate their impressions of MEA in domains of effectiveness, adoption, implementation, maintenance, usability and overall satisfaction. We will also solicit specific suggestions for improvement.
Primary outcome
The primary endpoint is the percentage of guideline-based recommendations acted on by clinicians for hyperlipidaemia, atrial fibrillation and heart failure. We hypothesise that care teams with access to MEA will provide more guideline-consistent care than care teams without access to MEA.
Secondary outcomes
Secondary endpoints include the following: (1) the percentage of previsit care process guideline based recommendations acted on by clinicians for each condition individually (ie, hyperlipidaemia, atrial fibrillation and heart failure); (2) the change in percentage of patients on optimal therapy for each condition from the initiation to the completion of the study; (3) self-reported efficiency in managing the three clinical conditions; (4) self-reported use of risk calculators; (5) overall use/burden of clinical decision support tools; (6) overall impressions of MEA in the domains of effectiveness, adoption, implementation, maintenance, usability and overall satisfaction and (7) overall use of the application.
Statistical considerations
Primary analysis
The study design provides two levels of hierarchy in the data. First, the unit of randomisation was the care team, which includes multiple clinicians. The second level of hierarchy is the individual clinician. For the purpose of the analysis, we expect that the variance component due to the care team to be negligible and potentially not estimable. Therefore, the primary endpoint will be tested with a generalised estimating equation (GEE) model using an identity link and binomial distribution with the clustering variable being the individual provider identification number. The primary parameter of interest in this model will be the main effect of the MEA intervention. With the identity link, the parameter estimate is interpretable as the change in percentage points of the primary outcome between groups. The clustering will be incorporated into model by means of the robust variance estimator using care team as the clustering indicator. As a sensitivity analysis, Rao-Cramer adjusted χ2 tests will be computed to compare the primary outcome between interventions. In addition, the primary outcome over time will be modelled using GEE to determine if temporal trends in the data are present. Secondary endpoints will be tested using a similar analytical strategy. For the primary analysis, we will assume that if a patient has more than one encounter in course of the study, that each of his or her encounters is statistically independent. This is to address the potential lack of identifiability of multiple encounters within patient due to the deidentification process as well as possible convergence issues with the higher order hierarchical model.
Sample size estimation
The sample size calculations are based on the following factors: (1) target power 80%; alpha=0.05 (two sided); no interim analysis/alpha spending function; (2) randomisation of clusters: approximately 10 provider teams (clusters) to MEA and approximately 10 provider teams (clusters) to standard of care; (3) an estimate that 2% of the patient encounters that are not exposed to the MEA will have a change in the care plan that would have been suggested by information directly reported by the MEA system had it been available; (4) an estimate that MEA will increase the percentage of patient encounters with a change in their care plan to 4% (this change is consistent with the median change seen in a systematic review of similar studies11) and (5) an intraclass correlation (ICC) range between 0.005 and 0.0075 based on the pilot study data. Based on these assumptions, the total number of patient encounters per cluster required is estimated to be between 263 (ICC=0.005) to 772 (ICC=0.0075). For study planning, the higher number is selected such that the total sample size per arm is 7720 (15 440 total patient encounters). Assuming each care team will have at least 20 patient encounters per clinic day, a minimum of 39 clinic days are expected for each team. We conservatively estimate that it will take approximately 6 months to provide some protection against the uncertainty of the ICC, variable staffing loads and vacation schedules over the summer and for sufficient time to monitor usage patterns over time. The total number of patient encounters during this period of time cannot be determined with certainty and it is expected to vary by care team. The institutional review board approval allows the protocol to use the first 50 000 patient encounters during the 6-month study period.
Ethics and dissemination
The clinicians were deemed human subjects and written informed consent for their participation was obtained. Health Insurance Portability and Accountability Act (HIPAA) authorisation for clinicians did not apply. A waiver of informed consent and of HIPAA authorisation was granted for the clinicians’ patients. All patient records that are used for research in the state of Minnesota must include research authorisation. Records for which the patient has declined authorisation were excluded from the study.
A waiver of informed consent for patients was granted by the institutional review board as the use of MEA was considered to have minimal risk to the patients and to not adversely affect their rights or welfare. Additionally, it was deemed that the research could not be practically conducted without waiver of consent as the tools could not run only on select patients for computational efficacy reasons and due to the size of the study. A waiver of HIPAA authorisation for patients was granted as: (1) it was deemed there was minimal risk to the privacy of individuals as the data will be deidentified by the IT team before being shared with the research team; (2) the research could not practicably be conducted without the waiver for the reasons stated above and (3) the research could not be conducted without access to and use of the protected health information, as the MEA computing algorithm uses protected health information in its algorithms.
Results of the study will be disseminated though publication, as well as meetings and newsletters targeted at key stakeholders.
Discussion
Our study will rigorously evaluate the effectiveness of an automated clinical decision support tool for improving adherence to best practices for patients with hyperlipidaemia, atrial fibrillation and heart failure. This notification system addresses many of the limitations of previous tools by integrating recommendations seamlessly into the EMR and by providing clinicians with both relevant patient data and actionable recommendations.
This builds on our earlier work in an inpatient setting that used a predecessor of MEA26 and illustrates the continued evolution of our clinical decision support system.27 If effective, this decision support tool could be scaled to address the management of other common chronic diseases.
Strengths and limitations
The strengths of this study are that the clinical decision support tool is integrated within the clinical workflow and was designed with iterative clinician input. The study enrols primary care clinicians from two specialties (family medicine and internal medicine) and evaluates the impact of the system on three distinct yet common clinical conditions. Further, the study rigorously tests the effectiveness of the tool through a randomised trial using objective and clinically meaningful endpoints (ie, changes in care to align with ideal management).
The main weakness of the study is that the findings may lack generalisability as MEA is a proprietary system, not yet available to unaffiliated institutions, and it was tested in an academic medical centre. Other limitations include our lack of tracking of outcome measures such as ED visits, hospitalisations or deaths, which during the study time frame would mainly be to ensure that the implementing MEA did not inadvertently lead to unintended negative consequences. We also did not stratify clinicians based on baseline adherence to the guidelines, which could lead to imbalances in the groups at baseline.
Supplementary file 2
![[SP2.jpg]](https://bmjopen.bmj.com/content/bmjopen/7/12/e019087/DC1/embed/inline-supplementary-material-1.jpg?download=true){kind=link}
Supplementary file 3
![[SP3.jpg]](https://bmjopen.bmj.com/content/bmjopen/7/12/e019087/DC2/embed/inline-supplementary-material-2.jpg?download=true){kind=link}
Supplementary file 1
![[SP1.jpg]](https://bmjopen.bmj.com/content/bmjopen/7/12/e019087/DC3/embed/inline-supplementary-material-3.jpg?download=true){kind=link}
References
Footnotes
Contributors REC and MES were involved in the development and validation of the clinical decision support tool. All authors except MS contributed to the study design. All authors are involved in the implementation of the project. MEK, REC and MRS wrote the first draft of the manuscript. All other authors revised it critically and gave final approval for publication.
Competing interests None declared.
Ethics approval Mayo Clinic Institutional Review Board.
Provenance and peer review Not commissioned; externally peer reviewed.