Article Text
PDF
Abstract
Objectives To determine whether an entirely electronic system can be used to capture both patient-reported outcomes (electronic Patient-Reported Outcome Measures, ePROMs) as well as clinician-validated diagnostic and complexity data in an elective surgical orthopaedic outpatient setting. To examine patients' experience of this system and factors impacting their experience.
Design Retrospective analysis of prospectively collected data.
Setting Single centre series. Outpatient clinics at an elective foot and ankle unit in the UK.
Participants All new adult patients attending elective orthopaedic outpatient clinics over a 32-month period.
Interventions All patients were invited to complete ePROMs prior to attending their outpatient appointment. At their appointment, those patients who had not completed ePROMs were offered the opportunity to complete it on a tablet device with technical support. Matched diagnostic and complexity data were captured by the treating consultant during the appointment.
Outcome measures Capture rates of patient-reported and clinician-reported data. All information and technology (IT) failures, language and disability barriers were captured. Patients were asked to rate their experience of using ePROMs. The scoring systems used included EQ-5D-5L, the Manchester-Oxford Foot Questionnaire (MOxFQ) and the Visual Analogue Scale (VAS) pain score.
Results Out of 2534 new patients, 2176 (85.9%) completed ePROMs, of whom 1090 (50.09%) completed ePROMs at home/work prior to their appointment. 31.5% used a mobile (smartphone/tablet) device. Clinician-reported data were captured on 2491 patients (98.3%). The mean patient experience score of using Patient-Reported Outcome Measures (PROMs) was 8.55±1.85 out of 10 and 666 patients (30.61%) left comments. Of patients leaving comments, 214 (32.13%) felt ePROMs did not adequately capture their symptoms and these patients had significantly lower patient experience scores (p<0.001).
Conclusions This study demonstrates the successful implementation of technology into a service improvement programme. Excellent capture rates of ePROMs and clinician-validated diagnostic data can be achieved within a National Health Service setting.
- AUDIT
- PROMS
- Electronic Patient Records
- Patient reported outcome measures
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
Large sample size with >85% data capture rate.
First study to report on routine capture of matched electronic Patient-Reported Outcome Measures (ePROMs) and clinician-validated contextual data.
Retrospective sampling.
Single specialist centre in one surgical specialty, which does not confirm that the findings can be generalised.
Introduction
Patient-reported outcome measures (PROMs) are being increasingly used in modern healthcare to assess the quality of services provided.1 They have become an important subjective measure of patient health status and minimise clinician bias.2 Assessing and achieving outcomes important to patients is associated with higher rates of patient satisfaction and helps patients feel health professionals are engaged with their concerns.3 ,4 In order to be effective, PROMs must be easy to read and understand, succinct and capture factors which are important to patients and clinicians.5–7
Since April 2009, PROMs have been routinely captured for all National Health Service (NHS) funded hip and knee replacements, groin hernia repairs and varicose vein surgeries.8 This represented a paradigm shift in assessing outcomes that mattered to patients as opposed to just mortality and morbidity.9 The NHS PROMs programme predominantly uses paper-based questionnaires to collect the data, at an annual cost of £825 000,10 but as the use of PROMs expands to other conditions and therapy areas, innovative data collection methods will be required to improve ease of administration, data capture rates and lower costs. Electronic systems may help in this regard.
In 2013, our unit began a pilot of a new system to capture PROMS through entirely electronic means. At the outset, the Trust Board raised concerns that data capture rates and quality may be adversely impacted by the introduction of a previously untested electronic system. Therefore, our research question was to determine whether an entirely electronic system can be used to capture patient-reported outcomes (electronic PROMs, ePROMs) as well as clinician-validated diagnostic and complexity data in an elective surgical orthopaedic outpatient setting. We also examined patients' experience of our electronic system and factors which affected their experience.
Methods
Design and population
This study was carried out in a specialist surgical orthopaedic clinic setting in the UK. Since 2013, ePROMs have been captured on all adult outpatients attending clinic. Data were stored on a secure local server on the trust premises. On first use of the system, patient consent is requested for their data to be used for the purposes of audit, research and analysis. This study was carried out as a pilot within the Foot & Ankle Unit of the Royal National Orthopaedic Hospital NHS Trust. This was a retrospective review of data and our Trust Research and Development (R&D) department reviewed this study and classified it as a service evaluation project, not requiring formal ethical approval.
Patients were asked to complete a set of electronic questionnaires on a customised software package (Amplitude Clinical Ltd, Droitwich, UK) (figure 1A). These included: a quality of life score (EuroQol 5-dimensions index (EQ-5D-5L)), and Visual Analogue Scale scores (EQ-VAS and VAS) (figure 1B). Condition-specific scores included the Manchester-Oxford Foot Questionnaire: pain, walking, social activity and index scores (MOxFQ-P, MOxFQ-W, MOxFQ-S, MOxFQ index).2 ,11 ,12 Prior to their appointment, patients were sent a letter, via post, inviting them to complete the questionnaires online and the letter included a web address. Those patients who did not complete the questionnaire prior to their appointment were given the opportunity to complete the ePROMs on a portable tablet device at clinic attendance, assisted by trained data capture personnel. Where the patient was accompanied by an interpreter, the data capture personnel allowed them to be involved in the process and recorded any difficulties that arose. Patients were asked to rate their experience of the system (the Patient Experience Index) on a scale of 0–10 (10 being ‘extremely satisfied’ and 0 being ‘extremely dissatisfied’). Patients were also provided an opportunity to leave feedback on their experience in the form of free-text comments.
(A) A photograph of the electronic Patient-Reported Outcome Measures (ePROMs) portal being used on a tablet device in the outpatient setting. (B) A photograph of a patient completing an ePROMs quality of life questionnaire.
When the patients entered the clinic room to see the consultant, the ePROMs were used as an adjunct to the clinical assessment. The clinician captured diagnosis and complexity data using a separate interface within the same outcomes system (Amplitude Clinical Ltd, Droitwich, UK). Diagnosis was captured using SNOMED Clinical Terms (SNOMED CT),13 and complexity data included referral source, comorbidities, surgical history, and psychosocial factors including smoking status. SNOMED CT is a comprehensive, multilingual clinical healthcare terminology which consists of a hierarchical, computer processable collection of medical terms that might be used by clinicians in everyday practice. It enables consistent representation of clinical content in electronic health records.13 The SNOMED CT used in this study were previously validated by the NHS Health & Social Care Information Centre (NHS HSCIC) supported by the British Orthopaedic Foot and Ankle Society (BOFAS).14 ,15
As a backup for information and technology (IT) failure, a paper based questionnaire was developed so that the data officers could transfer the data to the electronic system at a later date. In the pilot, we captured all episodes of IT system failure leading to loss of data; and the number of times disability or language barriers caused issues. Free-text patient feedback responses were allocated into one of two categories, ‘Issues with PROMS questionnaires’ (in that they do not adequately assess patient's symptoms) or ‘other’ feedback. ‘Other’ feedback included positive or negative comments on the electronic system, and other general comments.
Choice of PROMs
The EQ-5D-5L is a generic score consisting of five questions covering five domains: mobility, ability to self-care, activities of daily living, levels of pain and mental state. Each domain can be rated from 1 to 5. It is reported on a scale from 1.000 to −0.205, where a score of 1.000 represents a perfect health state and a score of 0.000 represents a health state considered equivalent to death. Scores lower than 0.000 are possible and represent a ‘worse than death’ health state.16 ,17 The EQ-VAS is a 0–100 scale where patients mark their overall health state on a visual scale: 0 represents the worst possible health state and 100 represents the best possible health state. We also use the VAS pain score, which is a similar scale assessing pain on a scale of 0 to 10: 0 is the best possible score (no pain) and 10 is the worst pain imaginable.
The MOxFQ is a foot and ankle specific questionnaire which assesses symptoms in three domains: walking/standing, pain, and impact on social activities. These scores are presented independently on a scale of 0 to 100: 0 represents the best possible health state and 100 the worst.2 ,12 The results can also be presented as a single summary index score, also from 0 to 100.12 It is the most validated foot and ankle scoring system and has been shown to have a larger effect size for foot and ankle conditions than more general scores.2 ,11 ,12 ,18
Statistical analysis
Descriptive statistics were used and statistical analysis was performed using SPSS 22.0 (IBM, New York, USA). The Shapiro-Wilk test for normality was performed and all data were confirmed to be parametric in nature. An analysis of variance (ANOVA) test was performed to analyse differences in PROMs between subgroups of patients, which were further analysed with a Tukey post hoc test. The significance of the observed difference in means of continuous data between patient subgroups was calculated using an independent samples Student's t test. Differences in frequency of categorical data among subgroups were examined using a χ2 test. Correlations were analysed using Pearson's product–moment correlation. Means are presented with an SD. A two-tailed p value of <0.05 was considered statistically significant.
Results
Between March 2013 and October 2015, a total of 2534 new patients attended our outpatient clinics. Clinician-reported data was captured for 2491 (98.3%) patients and ePROMs were captured in 2176 (85.9%) patients. Half the patients completed ePROMs prior to attending clinic (1090 patients, 50.09%). Figure 2 illustrates the breakdown of where ePROMs were completed. All 2176 patients completing PROMs gave informed consent for their data to be used and were included in further analysis. All patients completing ePROMs had clinician-reported data captured. There were no episodes of IT failure and therefore the back-up paper system did not need to be used. No language issues or issues resulting from disability arose during the pilot phase.
A pie chart of the location and device used by patients to complete the electronic Patient-Reported Outcome Measures (ePROMs).
The mean age of all patients completing ePROMs was 49.9±17.4 years (range 18–89). There were 1303 (59.88%) females and 873 (40.12%) males. The mean Patient Experience Index was 8.55±1.85 out of 10 (range 1–10). Pearson's product–moment correlation demonstrated a very weak correlation between older age and a lower Patient Experience Index across all patients (Pearson coefficient=−0.097, p<0.001). The mean quality of life and clinical scores are summarised in table 1.
Patient demographics and scores between patients leaving comments and those not leaving comments
Patient feedback comments
Six hundred and sixty-six (30.61%) patients elected to leave comments. Table 1 details the differences in mean age, gender, Patient Experience Index and ePROMs between patients leaving comments and those who did not. Patients leaving comments were older (p<0.001) and had a lower mean Patient Experience Index (p<0.001) than patients not leaving comments. Patients leaving comments also had better (lower) MOxFQ-P (p<0.001), MOxFQ-W (p=0.005), MOxFQ-S (p=0.001), MOxFQ index (p<0.001) and VAS (p<0.001) scores.
Of the 666 patients leaving comments, 214 (32.13%) felt that the questions, as asked, did not adequately capture their symptoms. For example, one patient commented: ‘Ankle/leg problems don't always have pain associated with them. I have a foot drop, which causes me problems, but there is no pain with it’. Table 2 describes the categories of comments left and the differences in age, gender distribution, Patient Experience Index and PROMs for each group. There was no difference in gender distribution between patients leaving comments in different categories. There was also no association between the type of comment left and diagnosis.
Patient demographics and ePROMs scores for different subgroups of patients
An ANOVA test revealed significant differences between groups in mean age (p<0.001), Patient Experience Index (p<0.001), EQ-5D-5L (p=0.005), MOxFQ-P (p=0.008) and MOxFQ index (p=0.038). Patients experiencing difficulty with technology were older than those leaving a positive feedback (difference of 10.58 years, p=0.001). Patients leaving positive comments had a higher mean Patient Experience Index of ePROMs than other groups leaving comments (p<0.001). Patients who felt their symptoms were inadequately assessed had lower (better) MOxFQ-P scores than those leaving positive comments (difference of 5.86 points, p=0.028).
A subgroup analysis was carried out of patients who felt their symptoms were inadequately assessed by PROMs to all other patients leaving comments. We found that patients who commented on the inadequacy of ePROMs to capture their symptoms were a mean of 3.72 years older (p=0.006), and had a mean overall PROMs experience index 1.41 points lower (p<0.001) than other patients leaving comments. This subgroup of patients did not have a statistically significant worse set of clinical scores.
The difference in mean ePROMs scores for patients who felt their symptoms were not adequately assessed by the ePROMs, compared with all other patients leaving comments, is shown in figure 3. A breakdown into the types of comments left by these patients can be seen in figure 4. The majority of patients (42.53%) felt that the PROMs questions were not specific enough to capture their individual symptoms.
Bar charts comparing the results of ePROMs between patients who found the scores inadequately assessed their symptoms against all other patients leaving comments. Bars illustrate 95% CIs and significance is denoted with an ‘*’. Patients who felt their symptoms were not adequately assessed by the PROMs reported lower (better) VAS, MOxFQ-P, MOxFQ-W and MOxFQ index scores. ePROMs, electronic Patient-Reported Outcome Measures; MOxFQ, Manchester-Oxford Foot Questionnaire; MOxFQ-P, MOxFQ pain; MOxFQ-S, MOxFQ social activity; MOxFQ-W, MOxFQ walking; PROMs, Patient-Reported Outcome Measures; VAS, Visual Analogue Scale.
A pie chart demonstrating the different reasons patients expressed to explain why their symptoms were not adequately assessed by electronic Patient-Reported Outcome Measures (ePROMs).
Discussion
To the best of our knowledge, this is the largest reported series of simultaneous and matched electronic capture of patient-reported data (ePROMs) and clinician-reported data for all new adult patients attending a busy NHS surgical outpatient clinic. Recording matched clinician data allows contextualised analysis of collected data, the absence of which masks the complexity of certain cases and the meaningfulness of the changes in outcome scores for groups of patients.19
Data were captured on all patients, whether they were managed operatively or non-operatively. We had an excellent capture rate of both clinician-reported data (98.3%) and ePROMs (85.9%) over a 32-month period and on 2534 consecutive patients in the clinics of three consultant orthopaedic surgeons. All patients were adults. The use of PROMS in the paediatric population is less well developed and was not addressed in this pilot. Previous series have looked at the use of electronic capture of data but have been on much smaller numbers of patients.20–22 Gurland et al20 trialled the use of tablet devices to capture ePROMs on 103 patients in a colorectal surgical clinic and reported an improvement in the PROMs capture rate from 25% (with a paper system) to 96%. Roberts et al21 reported a pilot study on elective orthopaedic new patients. When patients were asked, via the post only, to complete ePROMs at home, the completion rate was only 35% (26 patients). However, when patients were also offered the opportunity to fill out PROMs in clinic on a tablet device, the completion rate improved to 74% (23 patients).21 Palmen et al22 emailed 24 patients postoperatively asking them to complete ePROMs online and found poor completion rates of 33%. In contrast, we found a preappointment completion rate of 50.1% and achieved an overall completion rate of 85.9% with much larger numbers and put this down to the way in which our pilot was part of a complete service improvement agenda and not an isolated technology ‘add-on’. Indeed, technology cannot be used as a proxy to improve service and requires a process change to be built into any adoption process.23
At the outset of the project, the key issues we wanted to address included patient and clinician compliance; backup plans for IT failure; patients having difficulty with technology; meeting the needs of patients with disability; and how to manage language barriers. Over a 32-month period, we did not experience any data loss, nor observe any difficulty with barriers such as disability or language. This, of course, does not mean that these concerns are unimportant, but our experience does diminish the emphasis that needs to be placed on these issues in preference to establishing a scalable solution for data capture.
Electronic systems have been widely employed with great success in other industries. The airline industry, for example, moved to 100% electronic ticketing in 2008, with an estimated 10-fold reduction in costs.24 Recent trends also indicate that people increasingly use mobile devices for internet searches and access and that this is likely to increase in the future.25 The NHS is driving towards a paperless NHS,26 and thus ePROMs are likely to become increasingly widespread in their use. In our 32-month experience, half our patients completed ePROMs prior to attending clinic, using a variety of devices, one-third of which were smartphones or tablet devices. Patients' experience and feedback was very positive, even among older patients.
One of the reasons we believe that this pilot was successful was that at the outset, we put together a roll-out team, which included representation from: the clinical teams, the IT department, the research department, the information governance team and management. This was an essential step to ensure buy-in at all levels to the process change which was an essential part of the technology roll-out programme.
We began this initiative as a pilot in our foot and ankle department (which consists of 3 consultants) and employed a single data officer. The data officer also attended weekly team audit meetings to report on compliance so that any issues could be dealt with immediately. As the pilot expanded across other units within the Trust, we increased the number of data officers to four full-time employees at the NHS Band 2 pay scale.27 Their role is to support patient experience and ensure data capture across all orthopaedic clinics at two distinct geographic sites where over 19 000 new patients are seen annually. The importance of the data officers cannot be underestimated as they are available to troubleshoot any technical or technology issues at the time of occurrence. The investment in four full-time equivalent data officers cost the Trust ∼£70 000 (€90 000; $100 000). Given that our intention was to roll out our ePROMs system to all new patients of whom 19 000 are seen annually, we estimated that this would equate to a cost of ∼£3.60 (€4.65; $5.25) per patient. This figure fares favourably with the cost of administration of the NHS National PROMS, for which we currently spend ∼£5 (€6.45; $7.20) per patient. In addition, the national PROMS only captures PROMS preoperatively and 6 months postoperatively, but once the electronic system is in place, patients can be contacted at future dates by email at no additional cost. In addition, since an increasing proportion of patients will complete ePROMs at home in advance of their appointment, it is feasible that these costs might reduce in the future.
Electronic PROMS
Almost a third of patients leaving comments felt their symptoms were not adequately assessed by ePROMs. We believe this is the first report of patients' experience of using ePROMs to assess their symptoms outside of a validation study for the individual PROM itself.
The patients who felt ePROMs were inadequate in assessing their symptoms had lower ePROMs experience scores than those that did not feed back such commentary. We initially thought these patients may be outliers from our general patient population, but no evidence of this could be found when analysing their diagnosis or quality of life scores (EQ-5D-5L, EQ-VAS). In addition, their MOxFQ-P, MOxFQ-W, MOxFQ index and VAS scores were better than those of other patients. This may be a reflection of their symptoms being inadequately captured by these questions, which are pain focused and hence do not address patients whose symptoms involve disability but without pain. Interestingly, the MOxFQ-S scores of patients reporting ePROMs did not adequately capture their symptoms was similar to those of other patients, suggesting that despite having less pain, these patients' social life was just as impacted as other patients in our cohort.
Patients and surgeons may have differing opinions on which symptoms are most important.6 In the surgical foot and ankle population, previous authors have shown that patients are most concerned with improvement in pain, being able to walk sufficient distances and being able to return to desired footwear.7 Achieving outcomes important to patients, as assessed by PROMs, is associated with higher satisfaction rates.3 ,4 However, if patients feel that symptoms important to them are not being adequately assessed, this may result in dissatisfaction, which is reflected in our results.
Of the patients who felt their symptoms were inadequately assessed, a majority felt that the questions were not specific enough to capture their individual symptoms (42%); this appeared partly due to the wording of the MOxFQ, which asks about restrictions due to pain, and does not take into account patients who have disability but little pain. This group of patients also expressed dissatisfaction with the EQ-VAS, stating that despite their foot and ankle problems, they would otherwise rate their overall health as ‘good’, but were concerned this would give a misleading impression of their disability. Other patients indicated that the questions placed an emphasis on pain and this was not their main problem (19%), for example, patients with neuromuscular conditions. Some patients had pain from other sources which they found difficult to distinguish from their foot and ankle pain (13%), such as patients with sciatica. A proportion of patients felt unsure how best to report their pain as their symptoms varied with activity and analgesic use: it was commonly reported that pain could be almost completely controlled with analgesia, but they did not wish to be dependent on medication, and this was not assessed (14%). High functioning patients felt that the questions did not sufficiently capture their specific restrictions, such as the inability to participate in sports (12%). This is summarised in figure 4.
In most cases, the PROMs used do assess the areas of concern raised. However, patients may have still been dissatisfied as only a small proportion of these questions addressed symptoms that were important to them and they reported concerns that the overall score would not reflect the impact that their foot and ankle problem had on their lives. As would be expected, our findings support the notion that for patients to appreciate the system, it has to be relevant to their problem, address their concerns and be easy to use. Although we use a combination of well-validated scoring systems, a static questionnaire cannot assess all possible symptoms, particularly in a complex setting. Development of context-sensitive dynamic PROMs systems may provide a better way to assess a variety of systems without creating long, cumbersome questionnaires and research is being carried out to develop computer algorithmic systems to capture patient-reported symptoms which might have clinical utility in the future.28
Strengths and weaknesses
We believe this to be the largest series to document matched patient-reported and clinician-reported outcome measures in a busy NHS tertiary care setting. Weaknesses include that our review was retrospective and comments were not mandated. Aside from the quality of life data, the PROMS used were specialty specific and therefore whether our findings can be generalised across other centres and therapy areas is yet to be proven.
Possible shortcomings of PROMs include ceiling or floor effects.29 This study did not attempt to address the recognised limitations of existing PROMS and merely replicated a paper process of data capture using electronic means. The advantage of this approach is that if in the future the NHS mandates different outcome measures, these could easily be replaced on our system with minimal effort. In addition, this study pertains to adults only and no attempt has been made to apply our findings to the paediatric population.
Conclusion
We conclude that the introduction of ePROMs into a busy NHS outpatient setting is possible with good patient and clinician compliance, but requires investment in infrastructure, staff and careful engagement of all levels of management in the execution of service change. Patient engagement and experience was very positive and <6% of patients using the system reported difficulties.
ePROMs can work as an effective data capture tool and will most likely have an increasingly important role within the NHS as it moves towards a paperless system. We also found that up to one-third of patients leaving comments felt that ePROMs inadequately assessed their symptoms, which is an important consideration in selecting the correct outcome measures to best reflect and capture the outcomes that matter to our patients.
Acknowledgments
The authors would like to thank Mrs Saroj Patel, Director for IT at the RNOH, and the Trust Board for having the foresight to enable the introduction of this technology platform. They also thank the NHS HSCIC SNOMED CT programme and the British Orthopaedic Foot and Ankle Society (BOFAS) for their assistance in the development of the diagnostic coding system used in this pilot.
References
Footnotes
Contributors AJG conceived the idea for the paper, helped in its writing, captured data, and critically reviewed the manuscript. KM analysed the data, performed the statistical analysis and wrote the manuscript. OB and JT collated and categorised the data. NC and DS helped with the data collection and critically reviewed the paper.
Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Ethics approval Royal National Orthopaedic Hospital REC.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Screenshots of the system are available on request by contacting to the corresponding author at andy.goldberg@rnoh.nhs.uk. A video of the system in action is openly available to the public and can be found at: https://www.rnoh.nhs.uk/clinical-services/foot-ankle-lower-limb.