Article Text
PDF
Abstract
Objectives To evaluate the efficacy and acceptability of ‘Whitu: seven ways in seven days’, a well-being application (app) for young people.
Design Prospective randomised controlled trial of Whitu against waitlist control, with 45 participants in each arm.
Participants 90 New Zealand young people aged 16–30 recruited via a social media advertising campaign.
Setting Participants’ homes.
Interventions Developed during the COVID-19 pandemic, and refined from a prototype version that was evaluated during a smaller qualitative study, ‘Whitu: seven ways in seven days’ is a well-being app that, as its name suggests, contains seven modules to help young people (1) recognise and rate emotions, (2) learn relaxation and mindfulness, (3) practice self-compassion and (4) gratitude, (5) connect with others, (6) care for their physical health and (7) engage in goal-setting. It can be completed within a week or as desired.
Main outcome measures Primary outcomes were changes in well-being on the WHO 5-item Well-Being Index and Short Warwick-Edinburgh Mental Well-Being Scale. Secondary outcomes were changes in depression on the Centre for Epidemiological Studies Depression Scale, anxiety on the Generalised Anxiety Disorder 7-item Scale, self-compassion on the Self Compassion Scale-Short Form, stress on the 10-item Perceived Stress Scale, sleep on the single-item Sleep Quality Scale and user engagement on the end-user version of the Mobile Application Rating Scale and via qualitative feedback during an online survey. Outcomes were evaluated at baseline, 4 weeks (primary study endpoint) and 3 months, and analysed using linear mixed models with group, time and a group–time interaction.
Results At 4 weeks, participants in the Whitu group experienced significantly higher emotional (Mean difference (md) 13.19 (3.96 to 22.42); p=0.005) and mental (md 2.44 (0.27 to 4.61); p=0.027) well-being, self-compassion (md 0.56 (0.28 to 0.83); p<0.001) and sleep (md 1.13 (0.24 to 2.02); p=0.018), and significantly lower stress (md −4.69 (−7.61 to –1.76); p=0.002) and depression (md −5.34 (−10.14 to –0.53); p=0.030), compared with the waitlist controls. Group differences remained statistically significant at 3 months for all outcomes. Symptoms of anxiety were also lower in the intervention group at 4 weeks (p=0.096), with statistically significant differences at 3 months (md −2.31 (−4.54 to –0.08); p=0.042). Usability of Whitu was high (subjective ratings of 4.45 (0.72) and 4.38 (0.79) out of 5 at 4 weeks and 3 months, respectively) and qualitative feedback indicated individual and cultural acceptability of the app.
Conclusions Given the evolving psychological burden of the COVID-19 pandemic, Whitu could provide a clinically effective and scalable means of improving the well-being, mental health and resilience of young people. Replication of current findings with younger individuals and in other settings is planned.
Trial registration number Australian New Zealand Clinical Trials Registry (ACTRN12620000516987).
- COVID-19
- pandemic
- mental health
- mobile apps
- mHealth
- coping skills
- well-being
- adolescent
- young adult
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information. Deidentified, collated data from this study are presented in this paper. Individual data sets are not available for sharing as participants did not provide consent for this information to be shared. Any other details of the study procedure are available from the lead author on request (please email h.thabrew@auckland.ac.nz).
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
This randomised controlled trial was conducted with adequate power, a low drop-out rate and a small amount of missing data.
Key audiences of New Zealand Māori and Pacific young people were included.
Enrolment was limited to users over 16 years of age and there were fewer male participants.
Outcome measures were self-reported and there was no blinding of participants or researchers.
Introduction
The ‘invisible pandemic’ of psychological issues associated with COVID-19 is only beginning to be realised.1 2 Young people are particularly vulnerable to developing such issues due to pre-existing mental health challenges3 and lockdown-related disruption of their developmentally related needs.4 Within the past year, increased rates of mental distress,5 anxiety,6 depression7–9 and suicidal ideation10 have already been identified among young people in multiple countries. Additionally, those who have contracted COVID-19 have reported high rates of post-traumatic stress disorder.11 Long-term adverse health, academic and occupational consequences of these psychological issues are likely,3 7 12 13 especially in previously recognised subgroups with greater health needs.11 14 Despite increased demand for psychological support, access to face to face services has been significantly disrupted and delayed.15 16 Furthermore, evidence-based interventions for preventing and addressing psychological issues related to the pandemic are rare.17
Over the past decade, an increasing body of research has demonstrated the effectiveness of digital mental health interventions at improving the well-being and mental health of young people.18–20 This has led to some being recommended as first line treatments for conditions such as depression by the National Institute for Clinical Excellence in the UK.21 Given the frequency of smartphone use by young people,16 mobile health applications (apps) have particular appeal as a means of supporting young people to safely and conveniently learn and practice skills in the real world.15 16 18 19 However, out of over 20 000 available mobile health apps, very few have evidence of efficacy.22 Since the onset of the pandemic, the demand for mobile health apps has considerably increased23 and policymakers have recognised them as a widely disseminable means of improving immediate and longer-term well-being.24
Prior to the pandemic, New Zealand young people were experiencing high levels of mental distress, depression and the highest suicide rate among developed countries.25–28 Due to concerns about these issues becoming significantly worse in the context of mandated social distancing and repeated lockdowns, our research team rapidly developed an app to support the emotional well-being of this group, with special emphasis on the needs of young people of Māori and Pacific ethnicity who had always been disproportionately affected by mental health issues.15 16 ‘Whitu: seven ways in seven days’ (Whitu meaning seven in the NZ Māori language ‘Te Reo’) was based on a range of cognitive behavioural therapy (CBT), psychoeducation and positive psychology techniques previously shown to have efficacy in young people.15 16 18 The development of Whitu is discussed in more detail in our protocol paper.29 A small pilot trial (n=20) of the prototype app demonstrated statistically significant within-group improvements in well-being (p=0.021), anxiety (p=0.005), depression (p=0.031) and stress (p=0.004) between baseline and 6 weeks, but no significant changes in self-compassion, or sleep (in press, data available from the authors on request). User feedback led to improvements being made to the look and feel, cultural content and onboarding experience. This randomised controlled trial (RCT) was undertaken to evaluate the efficacy, usability and acceptability of the refined version of the app. We hypothesised that, compared with a waitlist control group, users of Whitu would experience improved well-being, self-compassion, sleep and reduced stress, anxiety and depression at 4 weeks and 3 months. Secondarily, we hypothesised that Whitu would be usable and acceptable to young people.
Methods
Study design
A mixed methods approach was used to determine the efficacy, usability and acceptability of ‘Whitu’. The study was prospectively registered with the Australian New Zealand Clinical Trials Registry.
Participants
New Zealand residents aged between 16 and 30 years who had reliable access to Wi-Fi, owned either an iPhone or Android mobile phone, were considered ‘healthy volunteers’ and not currently receiving mental health treatment, and could read and understand enough English to use the app via an online social media advertising campaign were recruited for the study. Participants were provided with a NZ $40 (Great British Pounds (GBP) 20) gift voucher on exit from the study as a thank you for their time.
Procedures
To optimise recruitment of New Zealand Māori and Pacific young people, the study was initially promoted to these groups via social media, and later opened up to individuals of any ethnicity. Participants (1) read study information, (2) completed informed consent procedures and baseline questionnaires and (3) were randomised to either the intervention group (Whitu app) or waitlist control group via REDCap, a secure web application designed to capture data for clinical research and projects that includes a randomisation module. At the point of recruitment, participants were asked not to use any well-being or mental health apps for the duration of the study. At the end of the study, they were also asked if they had done so, but none said that they had. Due to the nature of the study, neither participants nor researchers were blinded to treatment allocation. The intervention group was encouraged to download and use the app for 4 weeks. Both groups completed outcome measures via REDCap at 4 weeks and 3 months, following which control group participants were also provided with the app. No outcome measures were collected beyond this point. Further details are provided in our study protocol.29
Intervention
Whitu: seven ways in seven days is a free mobile application (app) that is currently available to New Zealand users via the App Store (https://apps.apple.com/nz/app/whitu/id1508135602?ign-mpt=uo%3D4) and Google Play Store (https://play.google.com/store/apps/details?id=com.carbonimagineering.whitu).
It contains seven positive psychology, CBT and psychoeducation-based modules that can be completed within a week. Users are encouraged to choose from a broad range of strategies and discover the ones that best work for them. Badge rewards and daily notifications encourage app completion and practice of preferred strategies. Further details of the app are provided in table 1 and figure 1. No user information or app analytic data are collected or stored over the Internet. Data entered by users are stored on their devices in an unencrypted SQLite database and can be safely removed at any time by deleting the app.
The seven modules of Whitu
Images of Whitu modules, including activities and badges.
Outcomes
Demographic data, including sex, age and ethnicity, were collected from all participants via REDCap at baseline. Outcome measures were assessed at baseline, 4 weeks and 3 months follow-up, with emotional and mental well-being outcomes at 4 weeks being the primary endpoints. Emotional well-being was measured using the 5-item WHO Well-Being Index (WHO-5).30 Mental well-being was measured by the 7-item Short Warwick-Edinburgh Mental Well-Being Scale.31 32 The scale has demonstrated good reliability (α=0.84) and validity in adolescent and young adult populations.33 34 Depression was measured by the 20-item Center for Epidemiological Studies Depression Scale (CES-D).35 The CES-D demonstrates high correlations with other depression measures and excellent internal consistency (α=0.85).35 Anxiety was measured by the Generalised Anxiety Disorder 7-item Scale.36 The scale has demonstrated excellent reliability (α=0.92) and validity in adults37 and adolescents.38 Self-compassion was measured by the Self-Compassion Scale-Short Form.39 The scale has demonstrated good reliability (α>0.86) in an adolescent sample.40 Stress was measured by the 10-item Perceived Stress Scale (PSS-10).41 42 The PSS-10 has demonstrated excellent psychometric properties compared with other stress measures, with good reliability and validity.43 Sleep quality was measured by the single-item Sleep Quality Scale (SQS).44 The SQS has been shown to have excellent concurrent and convergent validity with other lengthier sleep scales and has been demonstrated to be effective in determining clinically meaningful changes in sleep quality. User engagement was assessed by the app Subjective Quality subscale and the Perceived Impact subscale of the end-user version of the Mobile Application Rating Scale (uMARS) measure.45 The Subjective Quality subscale score consists of four items that determine user experience (eg, ‘Would you pay for this app?’). The Perceived Impact subscale score is derived from six items measuring the impact of using the app on knowledge, attitudes and intentions. The uMARS demonstrates good internal reliability (α=0.90), and the subscales demonstrate moderate reliability (α=0.71 and 0.80).45 In addition to the uMARS, participants also answered how many modules of the Whitu app they completed at each time point (1–7 modules) and provided brief qualitative feedback about their experience of using the app via an open-ended question in REDCap.
Data analysis
Using Gpower,46 we estimated a sample size of 90 participants (45 per treatment arm) would provide an effect size of f=0.155 for between group improvement in well-being using the WHO-5 index using a mixed analysis of variance (ANOVA) including within (three time points) and between (two groups) subject effects, with 90% power and at a two-sided significance level of 5%. This effect size relates to the between-group improvement in well-being found in a previous study of a web-based positive psychology intervention for mildly depressed adults.47 To ensure cultural acceptability of the app, we planned to recruit at least 36 (40%) young people of Māori and Pacific Island ethnicity. Baseline characteristics were summarised using means and SD or numbers and percentages. Repeated measures ANOVA was used with linear mixed models to include participants missing data at any of the three time points. The primary analysis aimed to determine whether changes in psychological outcomes were the result of the interaction between the intervention group and time, with post hoc tests to assess pairwise comparisons of groups at each time point and within-group changes over time. Cohen’s f2 was calculated as a measure of effect size for the group by time interaction.48 The primary comparisons of interest were between group differences at 4 weeks and 3 months, with results presented as marginal mean differences, 95% CIs and p values. Data were analysed using Stata software V.17, and statistical significance was set at p<0.05. Qualitative feedback was independently extracted and analysed by two authors (HT and ASS) using directed content analysis.49 Data was examined to the point of thematic saturation and any discrepancies in coding were resolved by consensus.
Patient and public involvement
Whitu was actively co-designed with New Zealand young people during the COVID-19 pandemic.29 However, no patients were involved in setting the research question or in developing plans for recruitment, design, implementation and dissemination of the results of the study.
Results
Participant characteristics
Of the 299 individuals who expressed interest, the first 90 eligible participants who met criteria were recruited to the study (45 per arm) between November 2020 and January 2021. One participant withdrew from the intervention arm without using the app due to technical difficulties or choice, four from the same arm were lost to follow-up at 4 weeks and another at 3 months. Only one participant was lost from the control arm at 4 weeks. Further details are presented in the Consolidated Standards for Reporting Trials (CONSORT) flow diagram (figure 2).
CONSORT flow diagram.
Participants ranged between 16 and 30 years, with a mean age of 23.8 years (SD 3.8). The majority of participants were female (n=79; 87.8%) and were students (n=59; 69.6%). Around a third reported having chronic health conditions including anorexia, anxiety, asthma, bipolar disorder, depression, eczema, epilepsy, hay-fever, hyperthyroidism, insomnia, migraines and polycystic ovarian syndrome. Participant demographics were similar between the intervention and control arm, apart from there being a greater proportion of participants reporting health conditions or medication use in the intervention arm and more participants of Pacific ethnicity in the waitlist arm. Further details are presented in table 2.
Participant demographics
Changes in outcome measures over time
Results presented in table 3 demonstrate that the intervention had a significant effect, as observed by a significant time by group interaction, on emotional (p=0.04) and mental (p=0.008) well-being, stress (p=0.001) and self-compassion (p=0.003). Measures of well-being and self-compassion were significantly higher and stress was significantly lower in the intervention group at both the 4-week and 3-month follow-up. The interaction between group and time on depression, anxiety and sleep did not reach statistical significance. However, differences between groups indicated evidence of better outcomes for those in the intervention group, with lower levels of depression (significant at both follow-ups) and anxiety (significant at 3 months) and higher sleep scores (significant at both follow-ups) being observed, compared with the waitlist controls. All outcome measures significantly improved over time within the intervention group (p<0.05; online supplemental table 1). There were no significant differences in outcome measures over time in the waitlist control group, except for sleep scores, which were higher at both follow-ups compared with baselines, although the effects were smaller compared with the intervention group (online supplemental table 1). Further details are presented in table 3, figure 3 and online supplemental table 1.
Supplemental material
Comparisons between groups in outcome measures over the study period
Marginal mean outcomes by group and study time point. CES-D, Centre for Epidemiological Studies Depression Scale; GAD-7, Generalised Anxiety Disorder 7-item scale; PSS-10, 10-item Perceived Stress Scale; SCS-SF, Self Compassion Scale-Short Form; SWEMWBS, Short Warwick-Edinburgh Mental Well-Being Scale; SQS, Sleep Quality Scale; WHO-5, WHO 5-item Well-Being Index.
User feedback
Overall, feedback regarding the app was positive, with special mention made by Māori young people regarding features designed to increase cultural appeal such as the introductory ‘karanga’ (welcome song). Participants expressed diverse, and non-culturally related preferences regarding individual modules, with newly learnt content being most valued. Suggestions for improvement included the use of shorter videos, improved navigation and greater flexibility with reminders (currently set at once per day). Six users with older mobile phones experienced some technical difficulties, but were still able to use the app. Key themes and examples of participant feedback are provided in table 4. Usability scores for Whitu are also provided in table 5.
Participant feedback
Usability for n=38 participants in the intervention group using the Whitu app*
Discussion
Overall findings
To our knowledge, this is the first RCT of a well-being app for young people undertaken during the COVID-19 pandemic and it addresses the clear gap in the COVID-related literature (ie, the lack of studies to address anticipated psychological effects of the pandemic) highlighted by Gilbody et al.50 Our results indicate that Whitu is an effective, usable and acceptable composite digital health intervention with which to improve multiple aspects of young people’s health including well-being, self-compassion and sleep, and to reduce anxiety, depression and stress. Benefits were evident at 4 weeks and sustained at 3 months follow-up. The fact that well-being in the intervention group actually improved during a pandemic is also clinically significant. Based on uMARS scores (table 4), usability of Whitu was high, and greater than that of recently developed mental health apps and established norms.51 52
Comparison with previous research
Our findings are consistent with recent review evidence that mindfulness and multi-component interventions are most effective at improving the well-being of clinical and non-clinical populations.53 Despite the potential floor effect with a non-clinical population, users of Whitu reported significantly improved symptoms of anxiety and depression. Resulting effect sizes were similar to the small to moderate effect sizes of individually targeted digital interventions for treating these conditions,54 suggesting that Whitu may be beneficial for clinical populations. Since the onset of the pandemic, a rapid review of existing digital mental health interventions has ascertained they are usable, safe, acceptable and likely to be effective in ameliorating at least some of the psychological consequences of lockdown.54 However, only one other RCT of a 4-week mindfulness-based intervention delivered to Chinese university students via Zoom and asynchronous WeChat video and audio recordings has actually been undertaken and shown to improve symptoms of anxiety and depression compared with technology-based social support.55
Given reports that only 3.9% of individuals who download health apps use them for a median of 15 days more than 2 weeks56 and that only 0.5%–28.7% actually complete them,57 the relatively high efficacy and acceptability of Whitu may be related to its intentionally time-limited design. Encouraging young people to learn new self-management strategies via the app and then practice them in the real world should also help with generalisation of these skills.57 Although some may argue that an app designed to support young people during the pandemic may be of limited chronological relevance, previous evidence from earthquake survivors in New Zealand suggests that psychological effects of major events are likely to be delayed, with rates of problems increasing by between 25% and 40% even after 2 years.58 59 Given the protracted nature of the current pandemic, its true psychological cost will only be obvious in retrospect.
Strengths and limitations
Strengths of this study include the adequate power, overall low drop-out rate (less than the typical drop-out rate of 25% during studies of other mobile health interventions)60 and small amount of missing data. In addition, given our desire to develop a culturally safe and relevant app, the appeal of Whitu to Māori and Pacific young people and its efficacy with these groups is reassuring and likely to reduce existing health inequities, thereby honouring New Zealand’s commitment to the Treaty of Waitangi.61 62 Weaknesses of the study include the lack of blinding of participants, inclusion of fewer male participants and use of self-reported outcome measures. It is also possible that group differences may have been smaller if an active control had been used instead of a waitlist control. As Whitu was designed to preserve well-being in the general population (rather than treat existing mental health issues) and in order to limit confounding from concurrent psychological therapies, inclusion in the study was limited to individuals not currently receiving mental health treatment. As such, its applicability to those already experiencing mental health issues remains unproven and further research with this group would be worthwhile. Around a third of participants reported having an existing health condition and this is in keeping with previous evidence that around 18% of New Zealand high school students and up to 45% of adults live with chronic health conditions.63 64 Although it is possible that individuals with pre-existing health issues were more likely to enrol in a study involving the use a new health app, the studied population appears to be representative of young people in the community. A greater proportion of participants dropped out from the intervention group than the control group and, although characteristics of those who dropped out and those who continued within each group were similar (please see table 1 and figure 1), our primary analysis may be biased by this missing data. For example, if reasons for dropout (which were unavailable) were related to worse outcomes, this might have potentially overstated the positive effects of the intervention. Although none of these individuals who dropped out provided feedback on their experience at the end of the study, this difference may reflect challenges in using, or lack of appeal of, eHealth interventions for some young people. Our results need to be replicated in other settings (such as schools) and with young people below 16 years of age to ensure their generalisability. Evaluation of Whitu’s efficacy with higher-risk groups such as young people with long-term physical conditions16 and more objective measures of app use and clinical outcomes would be valuable. Finally, future research would benefit from formal economic analysis to bridge the gap between researcher interests and policymakers.65
Conclusions
For the moment, this study provides preliminary evidence that Whitu is a clinically effective and scalable means of improving the well-being and mental health of young people during the COVID-19 pandemic.
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information. Deidentified, collated data from this study are presented in this paper. Individual data sets are not available for sharing as participants did not provide consent for this information to be shared. Any other details of the study procedure are available from the lead author on request (please email h.thabrew@auckland.ac.nz).
Ethics statements
Patient consent for publication
Ethics approval
This study involves human participants and was approved by University of Auckland Human Participant Ethics Committee (Reference 024542). Participants gave informed consent to participate in the study before taking part.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @HiranThabrew, @anna_boggiss, @SerlahiusAnna
Contributors HT and ASS conceived the research question. HT, ASS, ALB, DL, KS, EM, NC and AC designed the study. AC performed sample size calculations. HT and ASS applied for ethics approval and registration of the study. ALB, DL, KS, EM and NC undertook participant recruitment. ALB and DL set up and executed REDCap data collection. ASS, ALB and AC analysed quantitative data. HT and DL analysed qualitative data. EM and NC provided cultural oversight during the study. HT wrote the initial version of this manuscript and ASS, ALB, DL, KS, EM, NC and AC contributed to critical edits. All authors approved the final version of the manuscript. HT acts as the guarantor, accepts full responsibility for the work and attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.
Funding This study was generously funded by the Starship Foundation (SF 1562) and Auckland Medical Research Foundation (grant no 1720008), New Zealand. Funders did not have any direct involvement in the design or conduct of the study, data analysis or preparation of results. AS and HT came up with the concept for developing the Whitu well-being app. The IP for the app is owned by the University of Auckland and is not-for-profit.
Competing interests None declared.
Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.