Article Text
PDF
Abstract
Objectives Electronic patient self-Reporting of Adverse-events: Patient Information and aDvice (eRAPID) is an online system developed to support patient care during cancer treatment by improving the detection and management of treatment-related symptoms. Patients can complete symptom reports from home and receive severity-based self-management advice, including notifications to contact the hospital for severe symptoms. Patient data are available in electronic records for staff to review. Prior to the commencement of a randomised controlled trial (RCT), field testing of the intervention was undertaken to troubleshoot practical issues with intervention integration in clinical practice.
Design Observational clinical field testing.
Setting Medical oncology breast service in a UK cancer centre.
Participants 12 patients receiving chemotherapy for early breast cancer and 10 health professionals (oncologists and specialist nurses).
Intervention Patients were asked to use the eRAPID intervention and complete weekly online symptom reports during four cycles of chemotherapy. Clinical staff were invited to access and use patient data in clinical assessments.
Analysis Descriptive data on the frequency of online symptom report completion and severe symptom notifications were collated. Verbal and written feedback was collected from patients and staff and semistructured interviews were conducted to explore patient experiences. Interviews were transcribed and analysed thematically.
Results The testing ran from January 2014 to March 2014. Feedback from patients and staff was largely positive. Patients described eRAPID as ‘reassuring’ and ‘comforting’ and valued the tailored management advice. Several changes were made to refine eRAPID. In particular, improvement of the clinical notification, patient reminder systems and changes to patient and staff training.
Conclusions The field testing generated valuable results used to guide refinement of eRAPID prior to formal intervention evaluation. Feedback indicated that eRAPID has the potential to improve patients’ self-efficacy, knowledge and confidence with managing symptoms during treatment. A large-scale RCT is underway with data collection due to finish in October 2018.
- oncogenes
- patient reported outcome measures
- online intervention
- chemotherapy
- adverse events
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
The strengths of this field usability testing include the mixed methodological approach (combining qualitative and observational data) and the involvement of both patient and staff representatives.
This type of embedded clinical testing is vital for determining how new interventions work in practice.
The results are limited by the focus on one cancer group.
An ongoing randomised controlled trial will assess the impact of the Electronic patient self-Reporting of Adverse-events: Patient Information and aDvice online intervention on patients with cancer, oncology staff and health services.
Introduction
Systemic cancer treatment is associated with a range of side effects which can negatively impact patients’ quality of life (QOL) and become life threatening.1 As patients typically receive chemotherapy in outpatient settings, they are largely required to self-monitor symptoms at home. Patients can lack confidence in making decisions between obtaining clinical support or self-managing2 and can delay seeking medical advice,3 4 heightening the risk of symptom escalation and hospital admissions.5 Conversely subgroups of patients may routinely contact the hospital for reassurance in relation to mild side effects.2
There is growing evidence that the utilisation of patient-reported outcome measures during cancer treatment can aid the timely identification of physical and psychosocial needs, facilitate patient–doctor communication and assist decision-making.6–10 There has been a drive to develop electronic systems to allow remote real-time patient monitoring during cancer.11–13 Positive patient benefit (including QOL and survival) was recently reported in a US trial of an online system for metastatic cancer treatment.14
We developed an online intervention for supporting patient care during and after cancer treatment. Electronic patient self-Reporting of Adverse-events: Patient Information and aDvice (eRAPID)15 is web based and accessible from home or mobile device, for patients to complete symptom reports and receive severity-based advice. Recommendations for self-management are provided for milder symptoms and advice for when to contact the hospital for severe issues. A graphing feature allows patients to review personal symptom data over time. The system includes a facility to notify healthcare teams via email when a severe symptom is reported. Patient-reported data are transferred in real time to be accessed by health professionals through the hospital electronic patient records (EPRs) for use in routine consultations and assessments.
Following recommended usability principles of agile development,16 we involved staff and patient representatives throughout the eRAPID developmental processes, using an iterative design approach.17 Results from the technical usability testing are reported elsewhere.15 18 Here, we describe the field usability testing phase where staff and patients used eRAPID in a real-life clinical setting to troubleshoot practical issues not identified by standard usability testing19 20 and allow streamlining of eRAPID clinical integration prior to commencing a formal randomised controlled trial (RCT).21
Aims
For patients, the aims were to ensure:
System training was sufficient and feasible.
Routine (weekly) online symptom report completions were acceptable.
Self-management advice and severe symptom notifications were useful and appropriate.
For clinical staff, the aims were to ensure:
System training provided was sufficient and feasible.
Symptom report data were easy to access in the EPR and comprehensible.
Severe symptom notifications were correctly activated.
In addition, we wanted to assess the overall reliability of the Information Technology (IT) underpinning eRAPID.
Methods
Patient and public involvement
The eRAPID research programme has involved patient representation and collaboration at each stage from conception, funding application (involvement of a patient and public involvement coapplicant) and research delivery (membership on project management and steering committees). The eRAPID intervention was designed with substantial input from patient representatives from our dedicated Research Advisory Group (RAG). RAG members contributed to initial usability testing of the patient facing aspects of the IT system and advised on the content of the self-management advice.22 This paper describes the subsequent phase of usability work, where the intervention was taken for the first time for field testing in a clinical context with patients receiving chemotherapy and their associated care teams.
Clinical setting
The field usability testing was conducted in the breast medical oncology service at St James’s University Hospital, Leeds, UK. The Leeds Teaching Hospitals NHS Trust Research & Innovation Department approved the exercise as a service evaluation. Procedures were undertaken in line with Data Protection23 and Good Clinical Practice guidelines.24
Patient eligibility and identification
Patient eligibility criteria were (1) early breast cancer diagnosis, (2) starting at least four planned cycles of adjuvant/neoadjuvant systemic treatment, (3) internet access at home and (4) proficient English (to understand symptom assessments and self-management advice). Patients were identified by clinical staff and eRAPID was introduced by an oncologist or clinical nurse specialist (CNS). Interested patients were given an information sheet and passed to the eRAPID team for further information.
eRAPID demonstration and training
The eRAPID system is described in detail elsewhere.15 See figure 1 for a system overview.
Overview of eRAPID system illustrating the flow of data from inside and outside the hospital. EPR, electronic patient record; eRAPID, Electronic patient self-Reporting of Adverse-events: Patient Information and aDvice; NHS, National Health Service.
Patient symptom report and training
Participating patients received written information to take home and the researcher arranged to meet them at their first chemotherapy visit, where unique eRAPID login details were provided alongside a user manual. Patients were asked to complete the online eRAPID symptom report weekly (or more frequently if preferred) throughout four chemotherapy cycles (approximately 12 weeks). The online symptom report contained items from the locally devised patient-reported adverse event18 item bank based on the Common Terminology Criteria for Adverse Events grading system.25 Working in collaboration with the breast oncology staff, 12 core items were chosen for the main report (including pain, fatigue, physical activity, bowel function, sleep, temperature, chill, sore mouth and appetite). There was also the option for patients to add details on additional issues via a drop-down list of further symptoms and a free text option.
Clinical staff training
The breast CNSs were shown how to access eRAPID symptom report data in the EPR and given a one-page instruction sheet. They were also added to a mailing list to receive severe symptom notifications and encouraged to contact patients where feasible. Oncologists were trained on a needs-driven basis, receiving a demonstration and a one-page instruction sheet immediately prior to an eRAPID patient consultation.
Evaluation methods
Patient evaluation
Information on system acceptability and general feedback were collected through a number of methods:
The number of full symptom report completions and adherence to the weekly completion schedule.
Patients were provided with email and telephone contact details for the research team and a researcher met with the patient at their routine hospital appointments to check progress. The content of these communications was documented and collated.
Patients were asked to complete brief feedback in the user manual covering ease of system use and general comments or recommendations.
Patients were interviewed at the end of the testing. Semistructured interviews explored views on the technical practicalities, relevance/impact of the self-management advice and staff use of the reports and impact on medical management.
Clinical staff evaluation
Testing of the eRAPID system from the professionals’ perspective involved:
Completion of brief written feedback forms to record use of the eRAPID data and any impact on the consultation.
Direct observation of a subset of consultations where eRAPID information was available for staff. The researcher sat in the room and took field notes to describe how staff used eRAPID data.
Details of any severe symptom notifications sent to staff during the 12-week assessment were documented along with any action taken.
Ad hoc verbal feedback from staff was also documented by researchers throughout the 12-week assessment.
Evaluating the reliability of IT processes
Any IT issues reported by researchers, patients or staff during the assessment period were logged along with the action taken.
Iterative refinement of eRAPID
Throughout testing the research team collated feedback and identified issues were regularly fed back to the eRAPID project management team (consisting of oncologists, nurses, health informatics experts, patient representatives and researchers). The team decided how issues should be resolved and where eRAPID could be improved for the future RCT. A full report was prepared at the end of the testing, documenting all identified issues and actions taken.
Analysis
Patient and clinical staff evaluation
Descriptive accounts of the number of completed eRAPID symptoms reports were created along with the frequency of severe symptom notifications. Verbal feedback, comments from the user manuals, written feedback and notes taken during clinic observations were assimilated and categorised into themes.
Interviews were audio recorded, transcribed verbatim and managed in NVivo V.9. Initially a pragmatic approach was employed where any important issues raised in interviews were taken to the project management team to guide any immediate action. Interview data were subsequently fully coded and analysed thematically.26
Results
Participants
Patients
Testing took place between January and March 2014 with 12 patients (mean age=47.5 years, SD=10.3, range 33–73 years) (see figure 2).
Overview of patients approached and participating in the eRAPID field usability testing network. eRAPID, Electronic patient self-Reporting of Adverse-events: Patient Information and aDvice.
Clinical staff
Ten members of the breast care team participated including two adjuvant breast CNSs and eight doctors (four senior oncology consultants, four oncology trainees). Patients typically saw a CNS routinely throughout the 12 weeks and had one appointment with an oncologist before their third chemotherapy cycle.
Evaluation of eRAPID
Figure 3 shows a summary of data collected from staff and patients.
Summary of evaluation data collected from patients and staff. CNSs, clinical nurse specialists.
Compliance with weekly symptom reports
Over the testing period 104 full symptom reports were completed, 42% (5/12) of patients completed the report 11–13 times, 33% (4/12) completed 7–9 times and 25% (3/12) completed 4–6 times. Average adherence to weekly completion (ie, actual/expected completions per patient) was 63% (range 33%–92%).
Interviews revealed the most common reason for non-completion of the symptom report was simply forgetting. Most patients were in favour of a text or email reminder (n=8). Patients reported not completing the report when very unwell, stating it was not a priority. Others were unsure how often they should be completing. Feedback from the user manuals and interviews demonstrated that all the patients found eRAPID easy to use and did not report problems locating, logging in or using the system. A couple of minor suggestions were made for improvement, which were subsequently addressed (see table 1).
Summary of identified issues and actions taken to refine the intervention following field usability testing
Severe symptom notifications
Eight severe symptom notifications were activated, seven were not appropriate due to the framing of the symptom items which asked patients to report symptom experience within the last 7 days. This led to occasions where severe symptoms were being retrospectively reported with notifications activated for resolved problems. One patient found this experience alarming and stopped using eRAPID as a consequence.
It brought the alert up, and the hospital rang and thought I might possibly need an admission, I must admit that scared me a little bit…I said well no actually, these symptoms were a few days ago and now I’m absolutely fine… it were fantastic that they rang so quickly and I think it’s a great system for that, but I just thought oh no, I don’t want to go to hospital (Female, 44)
In addition, patients sometimes felt that notifications were not warranted for the symptom severity experienced, in particular, those for low physical activity:
It more or less panicked you a bit and said contact the hospital immediately… Because I still think I’d classed it correctly. It didn’t warrant an ambulance at the door or anything like that. (Female, 47)
However, the notifications worked well for the one patient who experienced an injection site reaction for several days. On completing the symptom report, she followed the advice to contact the hospital:
I wasn’t sure whether I should, so that advice was good. It will be good for people like that who are borderline. (Female, 73)
Following testing, the symptom report was refined to accommodate the notification issues identified (see table 1).
Thematic analysis of patient interviews
The following themes were identified:
Increasing knowledge and confidence
The self-management advice empowered patients by providing information and support to personally manage symptoms. Patients felt confident doing this, in the knowledge that the system provided a ‘safety net’:
It’s like a life line when you feel isolated when you’re at home and feeling poorly…you can have a lot of questions or problems regarding your illness and with one click they can be answered and absorbed within minutes. (Female, 49).
I would recommend it to anyone. It’s like a safety net for you and gives you the help to keep on going on through your treatment (Female, 73)
I think it does make you feel a bit happier—if you read something on there that says well, you will feel like this but you can do this, this and this… you’re happy with that (Female, 50)
Supporting decision-making
Patients felt that using eRAPID helped reduce their worry by aiding decision-making helping them feel more knowledgeable about when to self-manage and when to contact the hospital:
I’m a bit of a worrier and I think ‘shall, shan’t I, am I over-reacting?’ But I think that then would confirm to people that yeah, you should really ring the hospital so… it’s just like a little bit of extra home support isn’t it really? (Female, 33)
Coping strategy
Some patients found the symptom graphing feature useful for understanding patterns and this could be both reassuring and motivating.
For me personally, I just think I can’t do this anymore, I don’t like this, this is awful, and I find that I’m very disheartened and I can’t see an end to it……But then when you look at the graphs, you can think, but I did get better…and my mouth has got better, and my diarrhoea has stopped and… you can see that there is a pattern and that it will get better. It makes me feel better. (Female, 60)
Staff evaluation
Notifications for severe symptoms
Clinical staff agreed that notifications for retrospectively reported severe symptoms were not relevant and that some of the notifications sent for low physical activity were unwarranted. When this issue was identified, email notifications were redirected to the eRAPID research nurse for the remainder of the field testing, who liaised with patients and clinical staff where needed.
eRAPID patient symptom report data
The staff feedback forms indicated that symptom report data were easy to access and interpret and useful for identifying issues/problems for discussion (n=6), confirming knowledge of the patient’s problems, (n=5), providing additional information (n=5) and contributing to management (n=3). Several staff commented that it would assist interpretation of symptom reports if the EPR graphical displays included dates of chemotherapy delivery. In addition, staff commented that symptom information was most useful where patients had routinely provided reports throughout treatment.
The consultation observations confirmed that staff could easily access the symptom reports in the EPR but there were variations in utilisation. Some staff viewed the data but did not explicitly mention this to patients whereas others used it as a point of reference to guide the consultation and made this clear to patients.
Reliability of IT processes
The IT processes were largely stable. The notification system was reliable, with the patient and staff severe symptom notifications activated as expected. The patient symptom reports became temporarily unavailable to staff at one point. The problem was reported to the team and was resolved by the IT manager that day.
Refinement of eRAPID intervention and processes for integration
Following feedback from staff and patients, several improvements were made to streamline the integration of eRAPID into the clinical setting (table 1).
Discussion
The aim of this field usability testing was to observe end users (staff and patients) use of eRAPID in a real-life clinical setting in order to troubleshoot practical issues which may not be identified through standard usability testing.19 20 Feedback received from both patients and staff was positive and demonstrated the system was well received but also led to important modifications and improvements. The process allowed streamlining of intervention integration into clinical practice prior to formal evaluation.21
The majority of the notifications triggered for severe symptoms were for resolved symptoms patients reported retrospectively. This led to two key adjustments to the system prior to the RCT to avoid ‘false’ notifications and limit unnecessary patient worry and clinical burden. First, we added a branching question to allow patients reporting symptoms to provide further clarification on whether the symptom was ongoing or had been resolved. Second, the physical activity severity threshold was raised as both patients and clinical staff felt the original setting was too low. This was off-putting to patients and encouraging unwarranted hospital contact. For additional safeguarding, a further branching question was also added to this item to determine if patients reporting problems had help/support at home and assist with the identification of more vulnerable individuals.
Patients found eRAPID easy to use but many forgot to routinely complete the weekly report. As a consequence, the proposed text message/email reminder system was subsequently established. Patients reported valuing the self-management advice, particularly specific advice about when to contact the hospital and several patients described the system as ‘reassuring’ and a ‘safety net’. The testing highlighted the reciprocal relationship between patient and staff engagement in the system. Although staff felt the symptom reports were most valuable when routinely completed by patients, they did not always explicitly mention using the data in consultations. Moving forward in the RCT, we conducted a series of one-to-one and group training sessions with relevant staff involved in chemotherapy delivery and assessments (oncologists, CNSs, preassessment nursing teams). The sessions have included didactic elements (describing the eRAPID developmental work and the evidence supporting the use of using patient-reported data in clinical practice) and practical demonstrations of how to access patient-reported data. More recently an online training package was also created (accessible via a hyperlink in the EPR) which allows staff to view information as required. This online resource includes practical refreshers on accessing patient symptom reports along with interactive case studies that demonstrate how the data can be interpreted and used. There is emphasis within all the training formats on the importance of making overt reference to symptom reports with patients in clinical encounters to endorse the value of patient-reported data and encourage ongoing completions.
The interviews revealed the intervention could have the potential to increase patient self-efficacy and engagement with the management of their care. A recent systematic review has demonstrated the importance of self-efficacy in managing pain, symptoms and function in patients with cancer.27 In addition, high levels of patient activation (how engaged a patient is in their own healthcare) are associated with an array of improved health behaviours and health outcomes28–30 while lower levels of activation are associated with higher use of hospital resources.31
In the large-scale eRAPID RCT, the main outcomes focus on patient QOL and clinical process data (contacts with the hospital, emergency admissions) but we will also explore psychological variables that may help us more fully understand how patients can benefit from eRAPID.21 Specifically, we will explore the relationship between patients’ self-efficacy, patient activation32 and utilisation of the eRAPID system, self-management advice and symptom graphs. The WebChoice system in Norway has demonstrated that enabling patients to self-manage can be more beneficial for patients than symptom reporting alone.33
This field usability testing had some limitations. We only had the capacity (in terms of both staffing resources and time restrictions) to evaluate the system with patients with early breast cancer. These patient groups are relatively young compared with other adult cancer groups and are more likely to be digitally agile. However, internet access continues to increase34 and previous work has indicated that electronic systems are acceptable in other cancer groups.11 15 35–37 The RCT evaluates eRAPID in a broader population, specifically patients receiving chemotherapy for breast, gynaecology or colorectal cancer have been recruited from both adjuvant and metastatic treatment pathways.21
In summary, the field testing helped endorse the practical potential of eRAPID for supporting patient care but importantly uncovered issues which would not have been identified with standard usability testing alone. This was an invaluable exercise prior to the commencement of the ongoing RCT (data collection due to be completed in October 2018) which will evaluate the potential benefits of eRAPID for patients, staff and the National Health Service.
Acknowledgments
The authors would like to thank the staff and patients who have supported the eRAPID research programme to date, Leon Bamforth and Robert Carter for technical support and the following grant coapplicants/collaborators for their advice and leadership: Julia Brown, Jenny Hewison, Claire Hulme, Mark Conner, Jane Blazeby, Susan Davidson, Carolyn Morris (PPI representative), Peter Selby, Karen Henry, Liz Glidewell, Clare Harley, Geoff Hall, Martin Waugh and Kevin Franks.
References
Footnotes
LW and KA contributed equally.
Contributors All authors were involved in intervention development and design of field testing methodology. LW, AG and BC were responsible for patient recruitment and data collection. LW, AG, BC, KA, PH and GV all contributed to data analysis, interpretation of findings and manuscript preparation.
Funding This paper presents independent research funded by the National Institute for Health Research (NIHR) under its Programme Grants for Applied Research Programme (Reference Number RP-PG-0611-20008).
Disclaimer The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.
Competing interests None declared.
Patient consent Not required.
Ethics approval The Leeds Teaching Hospitals NHS Trust Research & Innovation Department approved this field testing as service evaluation. Procedures were undertaken in line with Data Protection and Good Clinical Practice guidelines.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Requests for data sharing should be directed to GV. Full interview transcripts are not available to protect participant anonymity.