Article Text
PDF
Abstract
Objective The dynamic and interactive mobile application Vett was designed to help change behaviour and is based on cognitive, motivational and visual techniques. Our aim is to investigate the acceptability, usability and utility of Vett as a personalised application for goal achievement.
Setting The trial took place at the rheumatology clinic at Diakonhjemmet Hospital, Oslo, Norway from January to June 2015.
Participants Twelve participants with osteoarthritis were recruited from a 3.5-hour multidisciplinary group-based educational programme (osteoarthritis school).
Interventions With the help of a physician, each participant followed a customised 12-week mixed-mode goal achievement plan with digital support based on preset goals, self-monitoring and individual feedback. Acceptability was measured as the perceived degree of goal achievement using a validated habit questionnaire scaled from 0 to 100. Utility and usability were assessed via 10 weekly questions and adherence by fulfilment of predetermined tasks.
Results Mean goal achievement was 73 (95% CI 68 to 78), an increase of 22 (95% Cl 17 to 26, p<0.01), which equals 48% improvement (95% CI 32% to 59%). Mean user satisfaction was 81 (95% CI 76 to 85), and technical usability was 80 (95% CI 75 to 84), which both increased during the study period.
Conclusion The high levels of acceptability, usability and utility support the feasibility of the personalised application Vett as a viable goal achievement tool.
- health informatics
- information management
- information technology
- rehabilitation medicine
- rheumatology
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
The digital and interactive training method Vett is developed to help change behaviour and is based on cognitive, motivational and visual techniques.
Acceptability was measured as perceived degree of goal achievement using the validated habit questionnaire Self-Report Habit Index.
Utility and usability were assessed by 10 weekly questions.
The feasibility outcomes should be interpreted with caution due to a small and not representative study selection, with no control group.
The participants may have been more motivated for behavioural change and more technologically fluent than others, which may have influenced perceived goal achievement, utility and usability rates.
Introduction
Chronic illness results in a substantial economic burden on individuals and society, but comes without a recipe for how to cope. As much as 30%–50% of chronically ill people do not follow treatment advice.1 WHO has stated that across diseases, adherence is the single most modifiable factor that comprises treatment outcomes, and the organisation calls for adherence-promoting technology.2 Given the constraint in the provision of care, there is a need to develop and assess the effectiveness of new treatment models.
Supporting long-term maintenance of desirable behavioural change is a major problem. To be able to complete tasks and achieve personal goals, requires coping skills. Studies show that many interventions are ineffective, except for cognitive behavioural techniques combined with goal planning, motivational interviewing, reminders and follow-up.3–5 Cognitive learning theory is central to behavioural change. In recent decades, one of neuroscience’s major discoveries was that not only does the brain map and synaptic connections change, but that these changes may also change the brain’s foundation for learning. Thus, cognitive training must be individualised and requires long-term efforts.6 7 Research shows that it takes an individual at least 6–8 weeks to change a single habit.8
The objective of goal setting is to create a change in patient behaviour. Such interventions have traditionally been delivered face to face, but today, the technology exists to empower the users to self-manage their challenges. One method for enhancing motivation into exercise and lifestyle changes is smartphone technology and special tailored applications based on preset goals, relevant exercises, self-monitoring and individual feedback. Health-related smartphone applications are increasingly being seen as viable and cost-effective solutions to improve treatment accessibility via mobile devices.9
Numerous health apps are available in the iTunes and Google Play stores, but very few have documented effects. The most reliable medical apps result from collaboration among developers, users and healthcare personnel, preferably conducted within the intended setting. Although mixed mode of delivery methodology in two meta-analyses is suggested to improve health outcomes and provide support for tailored over non-tailored interventions,10 11 user-centred design from the earliest exploratory stages is needed to help understand the needs, goals and preferences of the users. Most new health technologies are not products, but technology-enabled services that must offer meaningful benefits.12
The dynamic and interactive mobile application Vett, which was developed to help change behaviour, is based on cognitive, motivational and visual techniques. It is a mixed-mode behavioural change service based on preset goals, self-monitoring and individual feedback. Prior to the design of a clinical study to assess Vett’s ability to empower users to self-manage their challenges, user experiences need to be addressed to decide whether it meets the demands of future end-users. This proof-of-concept study aims to investigate the acceptability, usability and utility of Vett as a personalised application for goal achievement.
Methods
Participants and setting
The proof-of-concept study took place in Oslo, Norway from January to June 2015. With feasibility as its main purpose, a convenience sample of 12 participants with osteoarthritis was recruited from a 3.5-hour multidisciplinary group-based educational programme (osteoarthritis school) offered by the rheumatology clinic at Diakonhjemmet Hospital, Oslo. There was no randomisation procedure or control group. Eligibility criteria for the study were the diagnosis of osteoarthritis (classified within the ICD-10 codes M15–M19: polyosteoarthritis, osteoarthritis of hip, knee, first carpometacarpal joint and other unspecified osteoarthritis), a desire to apply smartphone technology to enhance motivation for exercise or lifestyle change and access to an iPhone or iPad.
Intervention
For this study, the already-existing digital and interactive training method, named Vett, was used. The application, which was developed to help change behaviour, is based on cognitive, motivational and visual techniques and has the following features: a mapping function of personal goals, tasks to perform and self-evaluation functionality. Vett is available at Google Play and iTunes stores and is designed as a tool to keep track of self-defined tasks to reach an individual goal. A mentor was assigned to each user. The application was connected to a server with an online administrative panel that supports multiple users, keeps track of the users’ progress, and enables dialogue between user and mentor.
All participants were offered a physician-aided and customised 12-week mixed-mode goal achievement plan with digital support based on preset goals, self-monitoring and individual feedback. One individual goal related to physical activity, weight reduction or stress-reduction combined with two or three corresponding weekly tasks, which were defined in the first consultation with the trial physician, were assigned, and the app was installed with personalised goals, cues and reminders. Both timing and reminder content is defined by the user. On entry into the study, background information on diagnosis, age, height, weight, quality of life (SF-36: short form (36) health survey) and educational level was collected. For feasibility purposes, the participants consented to provide weekly answers via telephone about perceived goal achievement and the functionality of Vett, as the purpose of the study was to assess the acceptability, usability and utility of Vett. A flow chart of the inclusion in the intervention group is shown in figure 1. As part of the methodology of the pilot, when the technological development was performed at the same time, we decided to contact all participants weekly about acceptability, usability and utility; weekly responses are shown in figures 2 and 3. All questions were answered on a scale from 0 to 100, where 0 represents complete disagreement and 100 complete agreement. In this pilot version of Vett, the application service users received individualised tasks as messages to which they were supposed to actively respond, by pressing the yes or no button when they were asked whether the task was completed.
Flow chart of the study.
Distribution of perceived goal achievement for each participant (n=12) of the study, by study week (illustrated as 12 consecutive circles) and as the average estimate. The average estimate is based on the 12 questions in the Self-Report Habit Index. Age, BMI, HE, date of study entry and goal characteristics are individually presented. BMI, body mass index; HE, higher education.
Distribution of utility as the mean value of six questions on user satisfaction and usability as the mean value of four technological feasibility questions by study week (n=11). The questions are listed in table 2.
Objectives, outcomes and feasibility criteria
The primary objective of the pilot was to assess the acceptability, usability and utility of face-to-face consultations combined with the smartphone application Vett based on preset goals, self-monitoring and individual feedback.
Primary outcomes were acceptability, utility and usability. Acceptability was measured as the perceived degree of goal achievement using a validated habit questionnaire, the Self-Report Habit Index.13 The 12-point index is composed of questions that assess three dimensions of habit formation: automaticity of behaviour, frequency of repetition and identity (table 1). Utility and usability were assessed with 10 weekly questions on overall satisfaction with Vett methodology and technical functionality (table 2). All questions, answered during phone interviews, were answered on a scale from 0 to 100, where 0 represented complete disagreement and 100 complete agreement with the statements with which they were presented. Perceived goal achievement was estimated as the difference between goal achievement at week 12 and baseline goal achievement (week 1), as an average of the 11 participants in the study. (One participant did not complete the study.)
Because the purpose of Vett is behavioural change and habit formation, perceived goal achievement was selected as a surrogate endpoint. Accordingly, an increase in perceived goal achievement of 20% or more was considered the most important criterion for assessing the success of feasibility. In addition, user satisfaction higher than 60% was also considered a valuable marker.
Statistical analysis and ethical consent
Stata V.14 and Excel 2010 were used for all analyses. Descriptive statistics (mean and SD) and numbers (per cent) were used to summarise sample characteristics and questionnaire responses and t-tests for group comparisons. All tests were two sided. P values <0.05 were considered statistically significant.
Patient and public involvement
Patients’ priorities, experience and preferences informed the research question and outcome measures for this study through previous experience from clinical patient follow-up performed by KJK. To enhance the acceptability of the study design, patient insight was collected before and during the study. Patients frequenting the Rheumatic school at Diakonhjemmet were specifically invited to volunteer for the study. The results of this study will be disseminated through the Norwegian Rheumatic association on a population level.
Results
Twelve participants were recruited for the trial that started in January 2015 and was completed by June. Individual 12-week mixed-mode goal achievement plans were consecutively set up in an outpatient physician-based setting. Except for one participant, who was lost in follow-up in week 2, all participants completed the study and participated in all weekly telephone interviews. There were 10 females and 2 males, with a mean age of 65 years (ranging from 61 to 70 years), and an average body mass index of 26 kg/m2 (ranging from 23 to 30). Six participants had osteoarthritis of the hip or knee, four of the hand or shoulder and two of the foot or ankle. The flow chart in figure 1 shows the inclusion of the intervention group.
Acceptability of Vett: self-defined goal achievement and habit formation
Figure 2 shows the individual distribution of self-defined goals, the goal achievement level at study entry and desired long-term fluency level. Individual goals were grouped into three categories: physical activity (n=7), weight loss (n=3) and stress reduction (n=2). Distribution of adherence to predefined tasks for each participant of the study by study week (illustrated as 12 consecutive circles) and the average estimate are also shown.
During the study period, perceived goal achievement increased in all participants. The mean increase was 22 (95% CI 17 to 26, p<0.01), which equals a 48% (95% CI 32% to 59%) increase in 12 weeks. Three participants completely reached their goals. Average goal achievement at week 12 was 73 (95% CI 68 to 78), compared with 52 (95% CI 46 to 57) at study entry. Table 1 shows the distribution of the 12 Self-Report Habit Index questions at weeks 1 and 12, respectively.
Distribution of average goal achievement with 95% CI at week 1 and week 12 and estimated difference between week 12 and week 1 (N=11)
The Self-Report Habit Questionnaire is composed of questions that assess three domains of habits: automaticity, frequency of repetition and identity. When goal achievement was estimated by the three habit subgroups, the automaticity domain improved by 45% (22, 95% Cl 18% to 33%, p<0.1), frequency of repetition improved by 42% (25, 95% Cl 18% to 33%, p<0.1) and the identity domain improved by 37% (20, 95% Cl 7% to 32%, p=0.006) (table 1).
Utility and usability: user satisfaction and technological feasibility
Utility estimates were based on 4 weekly user-satisfaction questions addressing motivation, simplicity, intuitiveness and ease of use, usability on six questions on tasks, reminders, feedback and a weekly summary (table 2).
Distribution of average responses with 95% CI to the six questions on user satisfaction and four technological feasibility questions at week 1 and week 12 and the difference between week 12 and week 1 (N=11)
Average usability by the end of the study was 80 (95% CI 75 to 84) and increased 32% during the trial with (19, 95% CI 7 to 30, p=0.04). The corresponding figure for user satisfaction of the service was 81% (95% CI 76% to 85%). Utility increased 13% (9, 95% Cl 5% to 14%, p=0.01). The week-by-week distribution showed that both utility and usability increased the first week, showed a declining tendency and stabilised around 80% after 7 weeks (figure 3).
Discussion
This study demonstrated that the physician-aided, customised mixed-mode goal achievement plan Vett, with digital support based on preset goals, self-monitoring and individual feedback, was feasible and well accepted. By the end of the 12-week study period, perceived goal achievement was high; on average, the individual goals were 50% closer to being achieved, and both user satisfaction and technical feasibility were reported at an 80% satisfactory level. The technological literacy in the sample can be considered as high, thus, despite the high average age, the users did not experience significant difficulties in using Vett. However, this may differ among user groups.
This study included a patient population where coping and lifestyle change is essential. Osteoarthritis is the most common joint disease, increasing with age and with no curative treatment.14 15 Treatment is education, exercise and weight reduction, which should be implemented in primary care for all patients at mild and moderate stages of the disease.16
The feasibility success criteria were fulfilled with ease, as goal achievement was increased by more than twice the a priori estimate of 20%. Although we did not design for a control group, 12 participants volunteered to define a goal to reach following the educational programme, rated their goal achievement level and allowed us to phone them after 12 weeks. Without exception, no change or improvement in their preset goal was made. This illustrates the difficulty to habituate physical routines.17 18 In addition, the fact that all participants were eager to continue using Vett after the pilot period was finished may also be interpreted as an informal endorsement of the usability of Vett.
Limitations and generalisability
Our study had several limitations. The feasibility outcomes should be interpreted with caution due to the small and not representative study selection, with no control group. The participants may have been more motivated to enact behavioural change and more technologically fluent than others, which may have influenced perceived goal achievement, utility and usability rates. Furthermore, the Self-Report Habit Index was used as a proxy for goal achievement and habit change. The index has proven to be reliable (Cronbach’s alpha >0.90), but it assumes that people are aware of reflections that they were not aware of when performing the desired behaviour.13 The index was originally designed on a 7-point Likert scale. However, for our telephone interviews, we decided to use a 1–100 scale for convenience, since index interpretation suggests that a habituation of proportionate strength takes place in the upper half of the scale.13
In view of the small sample, we are unable to comment on the generalisability of our findings. However, some interesting points emerged. Compliance with the weekly telephone interviews was excellent, and adherence to each individual predefined task was good throughout the study period. Where technical feasibility was concerned, the high usability rate of 80% was surprising, as we used a prototype of Vett that had several technical errors during the study period. A fully developed version of the technology is expected to improve usability.
Future developments
Vett has been further developed according to input from the pilot users. Based on user experience in the present study, including open feedback from the users, the beta version of Vett has incorporated new features to increase its user-friendliness. Technological stability of the interactive dialogue is now improved. Per the request of the pilot users, week-by-week graphical distribution of successfully performed tasks and a memo area for notes were added to increase motivation. Also, a feasibility study in the intended clinical setting was completed to make the design fit into the workflows and offer meaningful benefits. Following the feasibility study, a pragmatic trial is planned to examine both the effectiveness and implementation of Vett.
Conclusion
Chronic illness imparts a substantial burden on individuals and society and also comes without a recipe for how to cope. Technology-based tools have the potential to effectively promote behavioural change and increase adherence to the changes. The high acceptability, usability and utility in this trial support the feasibility of the personalised application Vett as a tool for goal achievement.
Acknowledgments
The authors thank all the staff at Diakonhjemmet Hospital who made it possible for us to recruit participants back to back in the completion of their 3.5-hour multidisciplinary group-based educational programme.
References
Footnotes
Patient consent for publication Obtained.
LNS and KJK contributed equally.
Contributors LNS and KJK conceived, designed and performed the experiments. LNS, KJK and AHP analysed the data. LNS and KJK authored the manuscript. AHP and JK critically reviewed the manuscript.
Funding This study was supported by The Norwegian Research Council (grant number:237766/O30).
Competing interests Except for the coauthors, KJK and JK have ownership in the firm m4Influence that invented the methodology and developed and built the pilot version of Vett. No other competing interests are declared.
Ethics approval Application for ethics approval was sent to the Regional Committee for Medical Research Ethics, Region Eastern Norway, but it was unnecessary.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement All available data can be obtained by contacting the corresponding author.