Using computerised decision support to improve compliance of cancer multidisciplinary meetings with evidence-based guidance
- Vivek Patkar1,2,3,
- Dionisio Acosta2,4,
- Tim Davidson1,
- Alison Jones1,3,
- John Fox5,
- Mohammed Keshtgar1,2
- 1The Breast Unit, Department of Surgery, Royal Free Hospital, London, UK
- 2Cancer Institute, University College London, London, UK
- 3Department of Oncology, Royal Free Hospital Hampstead NHS Trust, London, UK
- 4Centre for Health Informatics and Multiprofessional Education (CHIME), University College London, London, UK
- 5Department of Engineering Science, Oxford University, Oxford, UK
- Correspondence to Dr Mohammed Keshtgar;
- Received 19 January 2012
- Accepted 9 May 2012
- Published 25 June 2012
Objectives The cancer multidisciplinary team (MDT) meeting (MDM) is regarded as the best platform to reduce unwarranted variation in cancer care through evidence-compliant management. However, MDMs are often overburdened with many different agendas and hence struggle to achieve their full potential. The authors developed an interactive clinical decision support system called MATE (Multidisciplinary meeting Assistant and Treatment sElector) to facilitate explicit evidence-based decision making in the breast MDMs.
Design Audit study and a questionnaire survey.
Setting Breast multidisciplinary unit in a large secondary care teaching hospital.
Participants All members of the breast MDT at the Royal Free Hospital, London, were consulted during the process of MATE development and implementation. The emphasis was on acknowledging the clinical needs and practical constraints of the MDT and fitting the system around the team's workflow rather than the other way around. Delegates, who attended MATE workshop at the England Cancer Networks' Development Programme conference in March 2010, participated in the questionnaire survey.
Outcome measures The measures included evidence-compliant care, measured by adherence to clinical practice guidelines, and promoting research, measured by the patient identification rate for ongoing clinical trials.
Results MATE identified 61% more patients who were potentially eligible for recruitment into clinical trials than the MDT, and MATE recommendations demonstrated better concordance with clinical practice guideline than MDT recommendations (97% of MATE vs 93.2% of MDT; N=984). MATE is in routine use in breast MDMs at the Royal Free Hospital, London, and wider evaluations are being considered.
Conclusions Sophisticated decision support systems can enhance the conduct of MDMs in a way that is acceptable to and valued by the clinical team. Further rigorous evaluations are required to examine cost-effectiveness and measure the impact on patient outcomes. The decision support technology used in MATE is generic and if found useful can be applied across medicine.
How to improve the conduct of a cancer MDT and standardise decision making in accordance with best evidence.
Development and implementation of a novel clinical decision support (CDS) platform for breast cancer MDT.
This study evaluates (1) the concordance between the CDS suggestions and MDT recommendations and (2) the identification rate of potentially eligible patients for recruiting into the ongoing research trials, by the MDT and the CDS. A separate questionnaire survey was conducted at the national workshop at the Cancer Networks' Development Programme to get an estimate of acceptability of such MDT decision support systems by the cancer networks.
An advanced CDS platform could significantly improve the conduct of cancer MDMs.
Further robust evaluations are necessary.
Strengths and limitations of this study
We share our experience of developing an advanced decision support system and implementing it in a complex clinical environment of cancer MDT, which was subsequently adopted as a breast MDMs management tool.
The results reported here, however encouraging, are at this point indicative of the potential benefits but not yet conclusive. They should be treated with caution until further rigorous evaluations confirm the effectiveness and generalisability of the CDS system.
Unwarranted practice variation across different medical domains has unfortunately become a pervasive finding in health service research, and breast cancer care is no exception.1 A recently published study reported significant differences in breast cancer survival across hospitals in the same geographical region in England.2 The reasons for practice variation are multifactorial, and standardisation of care has been attempted by the introduction of Regional Cancer Networks in England and the adoption of the multidisciplinary team (MDT) model to promote maximal adoption of evidence-based practice. The MDT model is increasingly being adopted in other non-cancer medical domains, such as stroke, cardiovascular diseases and diabetes.
Many benefits of MDTs have been claimed, but few have been backed by strong evidence.3 ,4 However, despite a significant lack of prospective evidence, MDTs are well accepted in clinical practice; they are regarded as a major advance in management of patients with cancer and their use appears to be increasing.5 As many healthcare systems have already committed to and invested in the MDT model, further reductions in unwarranted variation are likely to be best achieved by improving their conduct and standardising their decision-making processes.6 Data collected by the UK national cancer peer review programme from over 1000 teams across six cancer types in England indicate that there is significant room for improvement in the conduct of MDT meetings (MDMs) and show considerable variability in the performance of MDTs.7 A recent national survey of more than 2000 members of cancer MDTs demonstrated agreement on the range of criteria necessary for effective MDT working.3 A review of the literature by the authors identified many pragmatic challenges and shortcomings in the current conduct of cancer MDMs summarised in box 1.8
Pragmatic challenges for cancer MDT meetings
Ensuring and documenting adherence with standards (eg, evidence-based guidelines).
Identifying patients who are eligible for recruitment into clinical trials.
Ensuring the consistent collection of crucial data, such as disease staging and outcomes.
Establishing robust mechanisms for prospective assessment of MDT performance.
Ensuring MDT recommendations are followed in practice.
Achieving the right balance of educational and care delivery objectives of this forum.
Establishing reliable interfaces with primary care to ensure continuity of care.
The Royal Free Hospital NHS Trust (RFH) serves a population of 2.6 million within the North London Cancer Network catchment area. The number of new patients (both benign and cancer) seen as outpatients by the breast unit in 2009–2010 was 2944. The Breast Cancer MDT at RFH was established in 2005, in line with the recommendations of the NHS Cancer Plan. The MDT uses a set of North London Cancer Network-approved clinical guidelines and a standardised minimum data set.
MDMs are held every week in a conventional conference format (figure 1). The core members of a breast MDT include breast surgeons, radiologists, pathologists, medical and clinical oncologists, plastic surgeons and breast clinical nurse specialists. A typical breast MDM discusses an average of 30–40 patients at various stages in their care pathways every week to decide further courses of action in their management.
Prior to the introduction of our computer-based service into the MDMs, an entirely paper-based record system was used to provide case summaries and to document the MDT's discussion and decisions. These records contained free (unstructured) text rather than coded and structured data. The trade-offs between structured (computer interpretable) and unstructured electronic health records (EHRs) are well known.9 Recording MDT discussions in an unstructured form, such as free-text clinical notes, scanned documents, pdfs, hinders the process of accurate measurement of MDT performance as computer-based data analysis and auditing tools cannot be used on unstructured data.
There are many commercially available information and communication systems, which can assist in the preparation, presentation and documentation of cases at the MDMs, such as EHR systems. However, the objectives of our MDT service improvement exercise was to go beyond improvements in data management by providing active support for evidence-based decision making, improving recruitment into clinical trials and supporting prospective audit.10
Measures of improvement
Evidence compliant care: adherence with clinical practice guidelines
With the increasing recognition of shortcomings in healthcare systems, there is a significant cultural and professional shift towards using evidence-based guidance. Evidence-based standards of care, such as published practice guidelines and technology assessment reports developed by authoritative organisations, provide an objective standard against which to assess MDT decisions. There is growing evidence that use of evidence-based guidelines can improve patient outcomes,11–13 and MDMs provide the best opportunity to actively promote an appropriate and judicious use of the guidelines at the point of care.
Promoting research: identification of patients eligible for ongoing research trials
It is widely accepted that recruiting patients into clinical trials is an effective strategy for ensuring that cancer patients get the best care as well as providing important information about the efficacy of treatments. However, the literature continues to report low rates of accrual to cancer clinical trials,14 and many organisations at national and international levels are investigating strategies for improving accrual rates. Cancer MDMs offer a major opportunity for identifying patients who are eligible for participation in clinical trials.15
In order to assess the performance of the breast MDT on the above-mentioned measures, we developed a computerised decision support system, MATE (Multidisciplinary meeting Assistant and Treatment sElector), that captures patient data, identifies eligible patients for clinical trials and suggests evidence-based treatment recommendations. MATE also captures MDT decisions and hence can automatically compare them with guideline recommendations.
We followed a systematic stepwise approach throughout the system development lifecycle. Requirements for MATE were identified through a systematic review of the literature8 and by working closely with members of the breast MDT at RFH. We adopted the common knowledge acquisition and design system (CommonKADS) methodology to develop a comprehensive process and knowledge model for breast cancer MDMs.16 A controlled vocabulary from the National Cancer Institute thesaurus17 was used to facilitate data standardisation. The evidence sources reviewed included clinical practice guidelines, systematic reviews and meta-analyses and reports of randomised controlled trials. Along with the guideline recommendations, the eligibility criteria of ongoing clinical trials in breast cancer that were open for recruitment at our institution were also coded into the system.
PROforma,18 an established decision modelling language for modelling clinical decisions and care pathways, was used to formalise decisions and supporting evidence in MATE. The PROforma language and application development software Tallis used in this project were originally developed at the Cancer Research UK. Tallis was used to implement a range of decision support and other servicesi as determined by the requirements development process outlined above and is used to update recommendations and other components of the PROforma knowledge base when new guidance is published. Tallis is being developed jointly by Oxford University and the Royal Free development team.
MATE functionality can be categorised under two broad headings: (1) structured data capture, presentation and audit, and (2) advanced evidence-based decision support.
Data capture: MATE allows users to capture detailed structured clinical data, including, demographics, comorbidities, test results, clinical findings, imaging, pathology and treatment-related data. The data are entered into the system either before (preparation phase) or during the MDMs (presentation phase). In the preparation phase, the data are entered by a clinician, who is responsible for the preparation of the meeting. Data entry is flexible, quick and secure, and it was found to reduce preparation time. If some of the test results such as pathology reports are not available before the MDT meeting, they can easily be entered in MATE during the meeting by a clinician without delaying the proceedings. MATE also provides patient summaries automatically and prospective audit facilities.
Advanced evidence-based decision support module: It is the key component of MATE that sets it apart from cancer tracking systems, EHR systems and the first-generation decision support, such as rule-based alert and reminder systems. MATE actively evaluates diagnostic markers histopathological data and other patient-related factors, such as co-morbidities to generate patient-specific recommendations for clinical management. The Tallis decision support technology enables MATE to rank the recommended options: for example, if the fitness of the patient is in question due to comorbidity, MATE can recommend the next best option with supporting evidence. In principle, patient preferences can also be factored into the MATE decision process and we are actively exploring ways of doing this in line with widely discussed needs for greater patient empowerment.
All clinical recommendations made by MATE are presented to the user together with a summary of the rationale in the form of arguments and supporting evidence. The MATE knowledge base has been developed with reference to a comprehensive set of published national and international clinical practice guidelines, which enables MATE to give recommendations even in complex cases that are covered by these guidelines.
MATE also provides quantitative risk estimates based on published models as an adjunct to the clinical recommendations.
Evaluation of concordance between MATE and MDT recommendations
MATE was used in the background to prospectively record the proceedings of breast MDMs between April 2008 and July 2009 to gather 1295 cases discussed in the MDMs during this period (each time a patient was discussed in the MDT meeting was counted as a separate encounter). The patient data and the MDT decisions were entered in MATE during the meeting by the first author. MATE recommendations were not shown to the MDT to avoid any confounding effect. After the meeting, the correctness of patient data and MDT recommendations entered in MATE were cross-checked with the official paper MDT records by a research associate from the research team and, in case of any discrepancies, the patient data and MDT decisions entered in MATE record data were amended to be in line with the official MDT record. Approval for an audit study was obtained from the Research and Development department of the hospital before starting the study, and data-security measures such as encryption were put in place.
One of the key features of MATE comparedwith a traditional EHR is the clinical decision support (CDS) element. MATE is able to actively evaluate patient data and to offer guideline-based recommendations in real time, which are specific for each individual patient. We compared MATE recommendations with the MDT decisions. The discordant cases (where MATE recommendations differed from those of MDT decisions) were further investigated by a panel who reviewed the patient's clinical notes. MATE also automatically flags patients who meet eligibility criteria for ongoing clinical trials.
Structured feedback from members of cancer networks in the UK
The MATE development team was invited to conduct a workshop at the England Cancer Networks' Development Programme conference in March 2010. The conference was attended by key members from all cancer networks, who are instrumental in governing and improving MDT conduct in their respective cancer networks. MATE was demonstrated in a workshop, and a questionnaire survey was conducted at the end of the presentation and discussion session. The aim of the structured feedback was to gather the views of the members of cancer networks about the usefulness of CDS systems in general and MATE in particular, in the context of cancers MDMS.
Respondents were asked to select from a choice of five categories (strongly agree, agree, neutral, disagree, strongly disagree) for five structured questions regarding usefulness of the system. They were also asked open-ended questions to find any perceived barriers and their general comments. For simplicity, we have combined ‘strongly agree’ and ‘agree’ responses into an overall ‘agree’ rating and ‘neutral’, ‘disagree’ and ‘strongly disagree’ responses into a an overall ‘disagree’ rating. The ‘neutral’ category was included in ‘disagree’ to ensure a conservative interpretation.
Evaluation phase results
The case mix of 1295 breast cases included cancers and benign pathologies. Table 1 shows the overall distribution of cases recorded on the MATE system during the study. Metastatic, recurrent and non-epithelial malignancies were excluded from the guideline concordance analysis as the guidelines and evidence-base for those subsets were not initially coded in MATE. In 239 cases of recurrent metastatic or non-epithelial malignancies, MATE therefore provided data capture services but no decision support. The remaining 1056 cases were analysed for concordance between management recommendations made by MATE and the actual MDT decisions; the level of concordance was encouragingly high (93.2%; N=984). When the 6.8% discordant cases were further analysed, it was found that in 3.2% cases, the MDT decisions that differed from MATE recommendations were corrected by the treating clinician in the results clinic.
MATE also identified 61% more patients who were potentially eligible for recruitment into clinical trials than the MDT alone. Note that MATE only screens the patients as possibly eligible for the trials, based on the main eligibility criteria. All the information needed before recruiting the patient is often not available to the MDT. Certain tests specific for the trial (eg, 2D Echo for ejection fraction) are done after MDT discussion, and the results are not available at the MDM.
Structured feedback results
The MATE workshop at the Cancer Networks' Development Programme conference was attended by 54 people, of whom 48 completed the questionnaire. The roles of respondents were categorised as follows:
Clinicians (Doctors and Nurses) = 13
Patients/survivors = 5
Service improvement managers = 18
Informaticians = 7
Others = 5
There was a very high consensus on the usefulness of CDS in general, and MATE in particular, for cancer MDMs. Most respondents (95.8%) agreed that CDS has a useful role in cancer MDMs. The majority of respondents found the services provided by MATE useful for the breast MDM (93.47) and potentially for other types of cancer MDMs (92.6%). The CDS component and ability to automatically screen patients for ongoing clinical trials were seen as the two most valuable capabilities of MATE by the majority of respondents (84.5% and 81.2% of respondents, respectively). Other capabilities of MATE, identified as valuable were patient data capture (70% of respondents), clinical audit services (67%), peer review support (58%) and education/training (45%). The majority of respondents (73.8%) were favourable to recommending MATE if it were made available in their network.
The survey also identified important barriers to large-scale deployment of MATE. The main perceived obstacle to adoption was double data entry (50%) in situations where existing data capture systems are in place, and it was suggested that MATE should be able to interface with existing data capture systems. Other barriers identified were costs and resources, clinical buy-in, scalability and the need for appropriate knowledge maintenance mechanisms that can cope with the large volumes of clinical evidence.
Challenges and next steps
We wish to emphasise that the role of MATE or any similar IT system is purely supportive and the MDT meeting continues to be led by the clinical team. Advanced IT systems can only complement an effective and functional MDT20 and cannot compensate for inherent weaknesses in team composition, organisation or operation. The preliminary audit results and the qualitative assessment data reported in this study, however encouraging, are at this point indicative of the potential benefits but not yet conclusive until further rigorous evaluations confirm the effectiveness and generalisability of MATE or similar services.
It has been reported that CDS systems produce better results when the developing team is also responsible for the trial of the system. One review reported, for example, that the success rate for CDS systems dropped from 74% to 28% when the systems were tested by independent teams.21 The team involved in the development of MATE was also involved in testing and the deployment of the system so replication of our results on other sites is a key objective. It was for the same reason that the questionnaire survey from the user was not conducted at this stage, and this is planned during the wider implementation phase. Demonstrating that MATE can confer significant benefits for other cancer MDTs is also a high priority. MATE has attracted the attention of the UK Department of Health's National Cancer Action Team, and deployment of the system in other NHS trusts is being explored.
Definitive evidence of the value of complex (multifaceted) interventions such as MATE requires a multicentre trial in which a cluster randomised design is likely to be the preferred methodology.22 The trial should look into all important impacts of the intervention, including quantitative measures of cost, patient outcomes and process measures as well as qualitative measures.
Patient involvement in decisions about their treatment is widely considered to be crucial to improving outcomes, and many cancer patients wish to play a more active role in their care. The current structure of the cancer MDT meeting makes patient participation very difficult to achieve.23 We are therefore exploring ways in which MATE could facilitate patient engagement, by extending access to certain of its functions by the patients. This could be achieved in a variety of settings, including consultations in results clinic and from the patient's home using the internet, allowing the patients to review their clinical history, the MDT recommendations and the reasons and justifying evidence for those recommendations.
We thank the Cancer Research UK for funding the development and the trial of the system and all members of the breast multidisciplinary team at Royal Free for their consistent support.
To cite: Patkar V, Acosta D, Davidson T, et al. Using computerised decision support to improve compliance of cancer multidisciplinary meetings with evidence-based guidance. BMJ Open 2012;2:e000439. doi:10.1136/bmjopen-2011-000439
Contributors All authors have made substantial contributions as follows: VP, JF and MK: conception and design; VP, DA, TD, AJ and MK: conducting the pilot study; VP and DA: developing the MATE knowledge base and software, and acquisition of data; VP and MK: analysis and interpretation of data; VP, DA, JF and MK: drafting the article. All authors: revising the manuscript critically for important intellectual content and have given final approval of the version to be published.
Funding The development and subsequent trial of the MATE decision support system was funded by Cancer Research UK by an extramural programme grant to JF.
Competing interest VP, DA, JF and MK have following specified non-financial interests that may be relevant to the submitted work. UCL (B) (a subsidiary of University College London) and ISIS innovation (a subsidiary of the University of Oxford) are actively seeking to commercialise aspects of this project through a spin-out company.
Ethics approval Ethics approval was not required for the study; however, for a randomised controlled trial, which is ongoing, ethics approval was obtained from the Moorfields and Whittington Research Ethics Committee.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement There are no additional data available.
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.