Article Text
PDF
Abstract
Objectives To establish patient and professional user satisfaction with the Advice & Interactive Messaging (AIM) for Health programme delivered using a mobile phone-based, simple telehealth intervention, ‘Florence’.
Design A service evaluation using data extracted from Florence and from a professional user electronic survey.
Setting 425 primary care practices across 31 Clinical Commissioning Groups in England.
Participants 3381 patients registered on 1 of 10 AIM protocols between March 2013 and January 2014 and 77 professional users.
Intervention The AIM programme offered 10 clinical protocols, in three broad groups: (1) hypertension diagnosis/monitoring, (2) medication reminders and (3) smoking cessation. Florence sent patients prompts to submit clinical information, educational messages and user satisfaction questions. Patient responses were reviewed by their primary healthcare providers.
Primary outcome measures Patients and professional user experiences of using AIM, and within this, Florence.
Results Patient activity using Florence was generally good at month 1 for the hypertension protocols (71–80%), but reduced over 2–3 months (31–60%). For the other protocols, patient activity was 0–39% at 3 months. Minimum target days of texting were met for half the hypertension protocols. 1707/2304 (74%) patients sent evaluative texts responded at least once. Among responders, agreement with the adapted friends and family statement generally exceeded preproject aspirations. Professional responders were generally positive or equivocal about the programme.
Conclusions Satisfaction with AIM appeared optimal when patients were carefully selected for the protocol; professional users were familiar with the system, the programme addressed a problem with the previous service delivery that was identified by users and users took an active approach to achieve clinical goals. However, there was a significant decrease in patients’ use of Florence over time. Future applications may be optimised by identifying and addressing reasons for the waning use of the service and enhancing support during implementation of the service.
- telehealth
- evaluation
- satisfaction
- primary health care
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
-
This service evaluation addresses the real-life use of a supported shared management programme delivered using simple telehealth technology deployed across a national population; thus, patients had not been ‘cherry picked’ which maximises generalisability of the results across primary care patients in England.
-
Quantitative data were gathered from both patient and professional users and was triangulated using patient activity data each month.
-
There was a significant quantity of missing data from failure to respond to user satisfaction questions, from intentional alterations to protocols by individual clinicians or Clinical Commissioning Groups such that user satisfaction questions were omitted and from patients stopping protocols early. Qualitative work with both patients and professionals would help to gain a richer understanding of what happened to these patients.
Background
Self-management support and patient education are core components of best-practice care for many conditions. Engaging people in keeping healthy and supported self-management are two of the eight priorities set out by The Kings Fund to transform individual involvement in health and care.1 However, interventions to support self-management often fail to show significant improvements in care.2 Indeed, after a pragmatic intervention failed to achieve significantly positive results due to lack of adoption of self-management resources into routine care, Kennedy et al3 suggested that “effective interventions are often not feasible and feasible interventions are often not effective.” This indicates that for new service delivery models to be effective, they have to be easy for primary care teams to adopt and create minimal change in practice workload and ideally, a net reduction. Mobile technology was suggested as a conduit through which self-management behaviours may be integrated into patients’ lives.3
Although evidence to date indicates that satisfaction with telehealth services can sometimes be good, substantive evidence that telehealth produces tangible outcomes, such as reduced hospital admissions, is lacking.4 However, it may be that such healthcare usage outcomes do not reflect the potential breadth of the value of telehealth services. Applications of such technology are wide, often with broader and less tangible value than simply preventing hospital admissions or reducing mortality/morbidity. For example, if used as an alternative mode of delivery of care to minimise barriers to seeking traditional methods of care (eg, the shift worker who cannot access the general practitioner (GP) to get their blood pressure (BP) measured), significant improvements in clinical outcomes of a population may not be observed but the intervention may reach patients who would otherwise not have been attended to. The involvement and enhanced responsibility patients experience through this type of service delivery may have longer term gains for the patient's health, well-being and future healthcare usage; for example, reassurance to carers and relatives may impact on health and social care use over time.4
‘Florence’ (or ‘Flo’) is a simple telehealth service that enables a patient to use their mobile phone to submit responses via text messages to a server that is periodically reviewed by their responsible clinician. This service was designed to support self-management and education using technology with which patients are already familiar; 92% of the population use a mobile phone, including 57% of people aged 75 years and older.5 To use Florence, patients are not required to have a specific make of phone or software installed, and they usually do not need any associated training and do not need to be computer literate.6 Assessment of clinical outcomes and changes in healthcare usage is necessary to prove the case for adoption of telehealth but, in order to ensure that services meet both patient and professional agendas, establishing the acceptability to both patients and clinical users is important too.7
Aims
The aim of this service evaluation was to establish patient and professional user satisfaction with the Advice & Interactive Messaging (AIM) for Health programme, delivered using a mobile phone-based, simple telehealth intervention.
Methods
The service
Following a local introduction of Florence, a service evaluation identified that patient satisfaction levels with the intervention were positive and there were some indicators that it may help to deliver BP management.8 ,9 Subsequently, the AIM programme was rolled out across England in March 2013 to help patients to take responsibility for the monitoring and shared management of their own condition, treatment or lifestyle. To this end, the AIM programme was initially launched with 10 clinical protocols which were delivered using Florence (see table 1 for the details of each protocol). All Clinical Commissioning Groups (CCGs) were invited to apply to be included in AIM and all those that applied were accepted. The rollout in the CCG began with an initial workshop organised by the CCG and delivered by the AIM team; the nature of the delegates attending varied depending on the CCG’s rollout plan, but usually included key CCG personnel who led telehealth or long-term conditions care, GPs and/or practice managers. The CCG rollout was usually facilitated by CCG staff and the AIM team working collaboratively with the expectation that a general practice team would become independently able to use Florence for its’ patient population. Patients were registered onto the Florence system by their general practice team; for hypertension and inhaler reminder protocols they were given shared management plans, which provided information about actions to take if the patient's readings and/or clinical condition deviated from normal. Practice teams were required to ensure that the shared management plans they adopted matched their own clinical protocols. Given that management plans used within AIM were designed to match usual care, the novel element of service delivery was the use of the simple telehealth system, Florence, to reinforce the information on the shared management plan by sending automatic responses detailing the actions patients should take in response to submitting readings outside of the acceptable range. The patients’ responsible clinicians periodically (eg, weekly) checked their submitted readings and contacted patients, if necessary, with further instructions.
Aims and success criteria for AIM programme and protocols
Success criteria
Prior to undertaking the national rollout of AIM, success criteria for the programme and each protocol were defined (see table 1 for success criteria relating to each protocol). Specifically relating to this evaluation are the programme aims, to enhance patient experience of shared management and to introduce CCGs and front-line practitioners in general practice to the everyday use of telehealth via Florence. In addition, due to the fact that patients could discontinue using the protocols if they wished, patient engagement was used as a proxy measure of patient satisfaction and attainment of minimum target days texting was defined within the success criteria, for hypertension protocols, and more broadly, by levels of activity each month.
Data collection
All patients who registered with Florence to use any one of the AIM protocols, between 1 March 2013 and 31 January 2014, were included in the service evaluation. Anonymous data relating to these patients and pertaining to the evaluation questions for each protocol that had been entered from registration to 30 April 2014, inclusive, were extracted from Florence. Data were excluded if the patient was not on an AIM protocol, if the patient started the protocol after 31 January 2014 or if readings were identified as being implausible or not from a real patient (see table 2 for definitions of plausible data). Data from Florence used in the evaluation were extracted using automatic data processing.
Definitions of (im)plausible data used in data extraction programming
To identify whether one of the primary aims of AIM, to enhance patient experience, had been met, Florence evaluation questions were sent at the end of each month of use (or week 1 for AIM01). For each protocol, the first question sent was an adapted version of the NHS friends and family test: Please text #1 if you agree with the statement ‘I would recommend this service to my family and friends’, or #2 if you disagree. Patients were not sent these questions if they had stopped using the protocol or if their responsible clinicians had altered the protocols on Florence to prevent evaluation questions being sent. Responses to this question were compared with the predefined success criteria (table 1) and used to evaluate patient user satisfaction with the AIM programme.
To establish the success with which front-line practitioners had been introduced to the Florence system, clinical user feedback was sought through the use of an anonymous electronic survey to clinical leads/champions (CL, clinicians appointed by the CCG who took a lead role in long-term conditions management), clinical telehealth facilitators (CTF, clinicians employed by their CCG to facilitate the rollout of AIM and supported practice teams) and clinician users (Clin, GPs and nurses in practices who participated in AIM). Five attitude statements relating to aims of the programme as a whole (healthcare usage, patient empowerment, improved clinical outcomes, popularity of Florence, cost-effectiveness) were presented to each professional group, associated with a five-point Likert scale, and respondents were asked to indicate their level of agreement. Respondents were also invited to provide free-text responses pertaining to what they thought had gone well and what they thought could be improved with AIM in general and/or individual protocols. An average of three professional CLs (n=93) from each of the participating CCGs were invited to participate in the electronic survey; these professionals had the option to disseminate the invitation more widely to any participating front-line practitioners. The survey was sent in summer 2013 and to others in spring 2014; feedback from both survey rounds was combined. Free-text data were extracted and developed into themes which were agreed by consensus within team.12 Following the data collection and thematic analysis, descriptive data analysis was undertaken using Microsoft Excel.
Results
Patient satisfaction
During the evaluation period, 3381 patients from 425 practices in 31 CCGs had registered onto 1 of the 10 AIM protocols. The majority of patients registered onto AIM01 (43%, n=1468). Patient activity appeared to be good at month 1 for the hypertension protocols (71–80%, 91% after 1 week for AIM01), and generally reduced over time (31–60% by protocol end; see online supplementary data file 1 for comprehensive usage data). The proportion of patients on hypertension protocols undertaking the minimum target days texting met the predefined 50% success criterion for AIM01 (1212/1468, 83%) and AIM02 (623/1114, 56%) but not AIM03 (32/208, 15%) or AIM10 (81/173, 47%; see online supplementary data file 1). For all non-hypertension protocols, patient activity at protocol end (3 months) was 0–39% (see online supplementary data file 1).
Of the patients sent evaluative texts, 1707/2304 (74%) responded at least once, which met the predefined 60% success criterion. In line with the activity data, proportions of patients responding to evaluation questions declined over 3 months. Overall patient users’ agreement with the adapted friends and family statement exceeded preproject aspirations (80% respondents agree (40% for smoking cessation protocols)) after each month of use. Across all active protocols at each time point, agreement with the adapted friends and family statement among respondents after month 1 ranged from 90% (AIM10) to 100% (AIM05 and AIM08), after month 2 from 56% (AIM03) to 100% (AIM05 and AIM08) and after month 3 from 79% (AIM03) to 100% (AIM05–07, AIM09). Satisfaction with individual protocols met preproject aspirations for all protocols at every time point except AIM03 at months 2 and 3 when 56% and 79% respondents, respectively, agreed with the adapted friends and family statement, and AIM08 at month 3 when there were no respondents. Although preproject aspirations were generally met, due to amendments to protocols and patients finishing protocols before evaluative texts were sent, respondents only represented 50% of those ever enrolled on any protocol. Therefore, online supplementary data file 2 provides a summary of the proportion registered on the protocol who were sent evaluative texts and the proportion of respondents who agreed with the adapted friends and family statement, presented both as a proportion of those sent texts and proportions of patients ever enrolled on a protocol.
Professional user satisfaction
Seventy-seven professional users responded in total over the two rounds. Of these, 60 were clinical users (GPs n=21, practice nurses n=17, healthcare assistants n=12, other n=10), 14 were CTFs and 3 were CLs.
Responses to attitude statements associated with five-point Likert scale indicate that 57–100% professional users generally agree that Florence is easy to use and, while 57% of all responders agreed that patient responses are regarded as accurate and only 7% disagreed with this statement (table 3). The majority of CTFs and CLs (56%) agreed that practices were keen to be involved, only one disagreed. Respondents were mainly equivocal (45%) or positive (35%) about whether Florence helped patients develop a greater understanding of their condition, medications and/or lifestyle choices. Responses were divided regarding whether or not Florence helps clinicians save time and whether adoption of Florence telehealth is cost-effective (table 3).
Professional user feedback to electronic survey attitude statements
Among free-text responses, 14/77 (18%) professional users did not give a negative comment, 8 (10%) did not provide a positive comment and 4 (5%) did not provide any free-text comments on what had gone well or what could be improved, primarily because they had insufficient experience of the programme at the time of the survey. Free-text feedback from the remaining 73 responders was summarised into six emergent themes (see table 4 for details of the content of each theme): (1) Florence system empowers patients and puts them in control of their health/condition, (2) use of appointments, (3) ease of use, (4) acceptability of the system, (5) acceptability of the protocol(s) and (6) support with using the system. Each theme contained a variety of positive and negative responses and often themes contained diametrically opposed responses about the same concept; for detailed responses see online supplementary data file 3. Hypertension protocols appear to have been particularly valued, especially AIM01. Three respondents (nurse n=1, pharmacist independent prescriber n=1, CTF n=1) suggested uses for Florence in the future. Some of these were the same or similar to protocols already in use, for example, smoking cessation, reminding patients to take their antihypertensive medication and monitoring those with hypertension. Novel ideas such as ordering prescriptions and twice yearly monitoring for BP for those on the combined oral contraceptives were suggested. Seven respondents also suggested other ways of improving Florence for ongoing use (GPs n=2, smoking adviser n=1, CTF n=3, CL n=1). Some of the users wanted information to be gathered about the time and money that could be saved by using this technology. Users felt that protocols could be improved by customising reminder times (eg, for shift workers), prompting patients to send readings at the time they are due rather than sending one message in the morning requesting both a.m. and p.m. readings, ensuring that the texts fitted closely with national guidelines, and to make sure that the number and type of texts being sent and required from patients is made very clear (and/or is adaptable) to pre-warn patients of the commitment required and/or to be able to optimise the protocols for the individual’s requirements. Further suggestions were made that clinicians and patients need to have robust advice on how to act in the event of a ‘breech’ (ie, a result falling outside of an accepted range) to minimise potential anxiety. Enhanced stakeholder engagement and feedback during development of protocols was also requested to improve acceptability, for example, the content of messages being sent.
Summary of the six themes emerging from free-text feedback from professional users
Discussion
The aim of this service evaluation was to establish the extent of patient and professional user satisfaction with the AIM programme which was delivered using a simple telehealth service and was used for a range of applications across a national primary care population. Overall satisfaction with the AIM programme appeared optimal when patients were carefully selected for the protocol; professional users were familiar with the system and the programme addressed a problem or gap in previous methods of service delivery that had been identified from the users. The majority of responding patients stated that they would recommend the service to their friends and family, and this was consistent across all protocols. Although the high patient satisfaction may not be surprising, as those who were unhappy with the programme were free to leave the protocol and/or may just have ignored texts from Florence requesting this information, this result is consistent with qualitative work that followed up an empirical study looking at the effect of self-monitoring and self-titration of medication for hypertension.13 Patient satisfaction responses should also be considered in the context of protocol completion which shows that popularity and usage of the protocols appeared to be strong at the end of the first month and then subsequently started to dwindle. However, not all non-responders necessarily represent those who were dissatisfied; protocols, particularly the reminder protocols, were designed to be supportive without the need to respond with data. Further, anecdotal feedback suggests that some patients struggled to find the ‘#’ button on their mobile phones. Therefore, the need for this special symbol may have prevented users from providing this feedback when prompted.
Feedback from professional users was mixed. Some professionals felt that self-monitoring improved the patients’ understanding of their condition and to take responsibility for their health which is consistent with empirical work in similar contexts.14–18 Although only 7% of professional respondents disagreed that responses submitted to Florence were accurate, free-text indicated a previously voiced15 concern about the quality of submitted readings. Professionals also voiced concern that the service increased anxiety among some patients. Again, this is not a novel concern16 and is one that has been substantiated by some patients.14 ,19 However, empirical telehealth studies that have specifically investigated anxiety have not identified increased anxiety among patients who are self-monitoring their BP compared with those undertaking more traditional care models.6 ,19 ,20 Anecdotal feedback suggested that some of the problems that professionals had with using AIM or, specifically, Florence were due to insufficient training or knowledge about how to use the system and problems with the local internet service. Such problems may reduce over time once routines and uses of the system become ingrained in everyday practice. Concern among professionals about patients’ ability to use the technology may be unfounded. Empirical work investigating the use of self-monitoring and telehealth in the management of chronic obstructive pulmonary disease showed that even those patients with little confidence in their ability to undertake the required activities, actually did manage the technology in the end.17
It is important to set some of the professional users’ comments in context, as it is possible some negativity reported may relate more to general dissatisfaction about telehealth and similar interventions rather than the AIM programme directly. This is because at the time the AIM programme was rolled out a government-led direct enhanced service (DES) was established to support GP practices to plan remote care monitoring for patients with stable long-term conditions to monitor their conditions so as to improve their quality of life and minimise visits to their GP practice.21 Originally this was introduced to allow plans to be realised in 2014/2015. AIM provided an approach by which practices could deliver this DES and allowed practices to familiarise themselves with the technology and available applications. However, rumours that the action stage of the DES planned for 2014/2015 had been withdrawn circulated at the end of 2013 and confirmation that the DES had been terminated occurred in March 2014. It is, therefore, possible that some clinicians had undertaken telehealth activities to match the DES requirements which were then shelved, and felt that their time and efforts had been wasted, particularly if they had not identified a need themselves for this service delivery method. Professional users may need to take a longer term view in investing time initially to become familiar with any new telehealth system in order to provide a slicker, more accessible service in the future.
Strengths and limitations
The value of evaluation through feedback has been recognised by The King's Fund as it was highlighted as one of the eight priorities to transform individual involvement in healthcare.1 This service evaluation provides a snapshot of the real-life use of AIM across a national population. Patients were not ‘cherry picked’; thus, the data from this evaluation are likely to be generalisable across the primary care population in England. Although the friends and family question is widely used across the UK health system, it may overestimate positive experiences and qualitative approaches may provide richer fidelity of information about the patient experience.22 As the data were gathered from both patient and professional users, recommendations made as a result of this evaluation are likely to meet both patient and clinical user priorities. However, due to the real rather than standardised empirical, nature of the AIM programme, missing data are an issue for some of the patient evaluative questions at some time points for some protocols. Evaluative texts were not sent if protocols were amended by patients’ clinicians, if CCGs chose to omit them from the protocols they were using or if patients stopped using the intervention early. Online supplementary data file 2 summarises the number of patients who were sent evaluative texts as a proportion of those ever on the protocol to contextualise the number of positive responders according to those ever registered on the protocol. Non-responding patients may include those dissatisfied with AIM.
To promote feedback from as many users as possible, the simple electronic survey was brief and therefore could not elicit specific details of all problems reported. It was occasionally not clear which element of the AIM programme the comments and responses related to. Response rates for the professional user survey could not be determined as an exact denominator could not be ascertained, thus an estimate of likely response bias cannot be made. Patient engagement with the service was used as a proxy measure of satisfaction in this service evaluation. Although it is recognised that satisfaction and engagement may not perfectly align, that is, a patient may be satisfied with but not engaged in healthcare services, we feel it is unlikely that a patient would be dissatisfied and actively engaged in this optional service. Therefore, although active patient engagement is likely to indicate positive patient satisfaction, we recognise there are limitations with this assumption.
Implications
For future service delivery in general practice
Taking into account no clear evidence of harm arising from this type of care, some evidence of benefits and the demonstrable patient satisfaction, simple telehealth appears to be an acceptable service delivery approach on a wide scale. However, to maximise engagement and satisfaction of both patients and professionals, it needs to be used for the right patients by the right professionals using the service in the right circumstances to address a problem identified by all users. Akin to other programmes, ‘one size does not fit all’14 ,17 and clinicians need to be selective when recruiting patients onto programmes, such as AIM, to ensure that they are compatible with the service and are receptive to and have confidence in remote management rather than traditional face-to-face care models. It may thus be wise to consider the components of the proposed theoretical framework of telemedicine acceptance based on established learning theory when implementing simple telehealth interventions.7 Based on this model, it is likely that clinicians who see the need and potential benefit of using the telehealth service will be more likely to select appropriate patients, interact effectively with individual patients and undertake proactive management. To overcome issues raised by clinical users, easily accessible support is required for practice teams while they familiarise themselves with the system, the associated documentation and the strategies employed by successful users of the system.7 ,23 Some professional respondents suggested future uses for Florence which included those that are in use or were similar to other protocols. This indicates the need for wider advertisement of potential protocols among active practices. Novel ideas such as ordering prescriptions and texted in BP readings twice a year for those on the combined oral contraceptives could be considered for future development. To prevent disengagement through frustration, irritation or perceptions of too frequent interactions, telehealth protocols need to be easily and widely adaptable to ensure that they are tailored to patients’ needs and the exact problem on which this service delivery method is focused. Responses requiring patients to use special keys or symptoms (ie, ‘#’) should be avoided
For future research
Uncertainties about how each protocol performed could be further evaluated through patient and clinical user focus groups. There is no way of knowing the extent to which patients who do not respond have benefitted from the programme nor what the barriers to providing feedback might be without initiating in-depth research and interviewing individual patients alongside review of their medical records. For example, what is the optimal frequency of messaging to ensure that patients remain engaged with and responsive to this type of service and are likely to follow advice to comply with medication, improve their health condition or lifestyle? Are patients happy to undertake clinical activity agreed to in their shared management plan and advised via Florence messaging, but do not wish to provide feedback? Further investigation is required to establish reasons for patients failing to complete the longer protocols, for example, AIM03, and the lack of engagement in the smoking protocols, for example, AIM06.
Conclusions
Although satisfaction with AIM appeared optimal when patients were carefully selected for the protocol, professional users were familiar with the system and the programme addressed a problem or gap in previous service delivery that was identified by users; there was a significant problem with patients’ reduced use of Florence over time. Future applications of the Florence programme or similar telehealth interventions may be optimised by considering the areas that caused difficulties for professional responders, specifically, providing choice of response methods, prompts for clinicians in the event of readings being submitted which are out of range and/or to periodically check the system, direct integration of results with electronic patient records and enhanced support during early implementation of the service. Further qualitative research may be of use to provide greater insights into the barriers faced by patient and professional users, and potential solutions and ideas for development in the future.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Data supplement 1 - Online supplement
- Data supplement 2 - Online supplement
- Data supplement 3 - Online supplement
Footnotes
-
Contributors RC, TC and PO conceived and designed the programme. EC designed the service evaluation in collaboration with RC, TC and PO. PO developed the data extraction protocols which were checked by EC. EC analysed the data with contribution from PO and TC. EC and RC conceived the paper. EC wrote the first draft of the paper. All authors contributed to the writing, review and revision of the paper, and all approved the final version before submission.
-
Funding The Department of Health funded the national rollout described. Mediaburst Limited, Stoke-on-Trent Clinical Commissioning Group's licensee funded the initial batch of text messages for each participating CCG.
-
Competing interests The Department of Health funded Stoke-on-Trent Primary Care Trust/then CCG to organise and lead the national rollout of Flo telehealth to 31 CCGs. Stoke-on-Trent CCG owns the intellectual property (IP) of Simple Telehealth including the trademarks Florence and Flo. The rollout project part funded a licence for Florence for each participating CCG from Stoke-on-Trent CCG's licensee. Stoke-on-Trent CCG receives royalties from its licensee; PO, the inventor of Simple Telehealth and Florence, receives royalties from Stoke-on-Trent CCG related to the licensing of the background IP.
-
Ethics approval This work evaluates a service improvement against predefined success criteria. It is not a research project; therefore, ethical approval was not required.
-
Provenance and peer review Not commissioned; externally peer reviewed.
-
Data sharing statement No additional data are available.