Participants

Fifty participants were recruited via media and social media channels, including the Black Dog Institute’s website and paid Facebook advertisements that targeted potential participants and their parents between April and June 2017. A sample size of 50 was selected in order to successfully meet the study’s feasibility and acceptability aims. Inclusion criteria were: aged 12–16 years, presence of at least mild insomnia, operationalised by endorsement of at least one of the following symptoms over the preceding 2-week period: difficulty falling asleep, difficulty staying asleep or waking up too early. These items are the first three questions on the Insomnia Severity Index (ISI),24 and were chosen to include a participant group with at least mild levels of insomnia. For study inclusion, participants also needed to own a smartphone running iOS or Android, have a valid email address, access to the internet and be able to provide personal and parental consent.

Measures

Intervention: ‘Sleep Ninja’

The Sleep Ninja app was derived from CBT-I and developed by our team, as a fully automated smartphone app. A participatory design process was used whereby young people contributed to the content, functionality and accessibility/user experience of the app through a series of focus groups.39 The core strategies included in the app were: psychoeducation, stimulus control, sleep hygiene and sleep-focused cognitive therapy. Sleep restriction, which aims to increase SE by reducing the amount of time spent in bed, was deliberately omitted because some support (parental and/or professional) is likely to be required to successfully implement sleep restriction, particularly in young people. Although sleep restriction did not comprise part of the app, there was instead a focus on the importance of regular sleep-wake cycles. The app teaches users about the importance of consistent sleep and wake times, and recommended bedtimes are calculated based on the time they need to wake up (according to sleep guidelines). This strategy draws from transdiagnostic approaches to target sleep difficulties that go beyond insomnia (eg, delayed sleep phase and irregular sleep presentations40) and may, therefore, be useful to adolescents experiencing a broad range of sleep difficulties.

The structure of the Sleep Ninja app includes six training sessions (lessons), a sleep tracking function, recommended bedtimes based on sleep guidelines, reminders to start a wind-down routine each night, a series of sleep tips and general information about sleep. The home screen has three options: train, track, and more (see figure 1). Users complete training sessions which are delivered through a chat-bot format where the sleep ninja essentially acts as a sleep coach. Training sessions take approximately 5–10 min to complete, and cover: (1) psychoeducation, information about circadian rhythms and the importance of keeping regular sleep schedules; (2) stimulus control, the value of only going to sleep when tired, and strategies that can be used at night when having trouble sleeping; (3) basic sleep hygiene such as avoiding caffeine and stimulating activity in the evenings, suggestions for daytime activities to promote night-time sleep (eg, exercise, no napping); (4) identifying and planning for high-risk situations, how to get back on track after a late-night or sleep in; (5) cognitive therapy including how to deal with unhelpful thoughts that can prevent falling asleep as well as sleep-related cognitive distortions, and (6) a final review session which summarises all of the material contained in the app. The user interacts with the app through a forced choice chat-bot format which is responsive to the input of the user, meaning it personalises information and recommendations based on the selections and sleep profile recorded by the participant. Users level up and reach their next ‘belt’ by completing one training session and tracking their sleep for three nights (out of a 7-night period). As there were six training sessions to complete, the app was made available for 6 weeks (42 days) before it locked. Users finish the programme with a black belt in sleep.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Example screens from the Sleep Ninja app. From left: homescreen, training session access and progress record, tracking and bedtime setting, more information.

It is notable that evidence suggests that the use of screens at bedtime interferes with sleep.41 Drawing on data from large epidemiological studies suggesting that refraining from screen use for the hour prior to bedtime alleviates potential interference from screens,42 this app has been designed to be used during the day. Users receive a prompt 1 hour before bed (calculated according to sleep guidelines and their wake-up time) to commence their prebed time routine and are encouraged to stop using electronic devices after this time. In fact, part of the cognitive component of the app is to educate and challenge beliefs about the importance of night time phone use in order to promote healthy sleep habits. We expect these factors to mitigate the risk that smartphone use in this context will contribute to poor sleep.

Procedure

Participants were encouraged to download consent forms if they met the eligibility criteria listed on the study website and submit this directly to the research team, once completed. Those who provided written informed consent and that of a parent or guardian were then enrolled in the trial and invited to complete the screening questions to verify study eligibility, before completing baseline questionnaires which included: demographics, ISI, PSQI, PHQ-A, GAD-7 and Expectation of Success. Participants could then access the first day of the online 7-day sleep diary. Another diary entry became available each day for the following 6 days and participants were reminded to complete entries via text message. At the completion of seven consecutive entries in the diary, participants were given access to the Sleep Ninja app on their personal smartphone devices. Participants could use the app for 6 weeks before the poststudy questionnaire was made available, which included the same battery as baseline with the omission of the Expectations of Success questionnaire and with the addition of the Acceptability and Reasons for Non-Adherence questionnaires. Participants then completed another 7-day sleep diary, which was delivered in the same format and schedule as baseline. After the study had finished, participants were invited to participate in a face-to-face or telephone interview to provide feedback on their experience of participating in the study. Participants were reimbursed for their time with gift cards to the value of AUD$10 each for completing baseline and poststudy assessment schedules; AUD$20 for a telephone interview and AUD$30 for a face-to-face interview.

Participant involvement and consultation

Prior to this study, a separate group of young people were consulted in a series of focus groups to inform the design, features and structure of the app (for more information please see39). As a feasibility and acceptability study, participants were asked to report on their experiences with respect to both the app and the study procedures, via questionnaires and an in-depth semistructured interview. Given that a key objective of this study was to assess the acceptability of the Sleep Ninja app, participants’ perspectives were of critical importance. A one-page lay summary of the study results has been sent to all participants.

Statistical analyses

Statistical significance was set at α=0.05. Summary scores for sleep diary variables at baseline and poststudy were obtained by averaging sleep diary entries at baseline, and averaging sleep diary entries at poststudy, respectively. All questionnaire and sleep diary variables were initially screened for excessive skew (>3) or kurtosis (>843). Six sleep diary variables did not pass screening and were further scrutinised (baseline: WASO entry; poststudy: time fallen asleep, TIB after final morning wake up, SOL, TST, SE variables). Examination of these six variables revealed each included an entry that was of an extreme value (z-scores ranged from |4.11| to |6.05|) and a decision was made to remove these six values (hence, n=47 for baseline WASO; ns=28 for postintervention time fallen asleep, TIB after final morning wake up, SOL, TST, and SE). Subsequently, all variables had satisfactory skew and kurtosis.

Questionnaire and sleep diary variables were examined using multilevel modelling. This modelling approach handles missing data by incorporating all available data from each subject into the analysis. Given the aims of our study, our interest centred on the main effect of time (ie, change from baseline to post). Random effects were modelled for intercept and time. Models were respecified with a random effect for intercept only in cases where there was no variation in individual baseline to poststudy changes. Within-group effect sizes were computed as the modelled mean difference between baseline and poststudy divided by the sample SD at baseline.

Baseline characteristics

See table 1 for characteristics of the study sample. Participants had a mean age of 13.71 (SD=1.35), were spread across school grade and nearly all were born in Australia and living in the city. Most participants reported difficulty falling asleep, with about half also reporting problems staying asleep or waking up too early, and about a quarter of the sample were receiving treatment for sleep or mental health problem.

Recruitment rate

There were more than 300 enquires made to the research team about participation in this trial. Of these, 60 individuals indicated eligibility and returned consent forms. Ten of these participants were not enrolled in the trial; four did not meet inclusion criteria and six withdrew prior to the trial. Reasons for withdrawing were: a change of mind (n=1), a lack of time (n=2), considering participation a chore (n=1), confidentiality concerns (n=1). One participant did not provide a reason. Therefore, 89% (n=50) of the 56 young people who provided consent and met screening criteria continued to trial. See figure 2 Consolidated Standards of Reporting Trials diagram for details.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Participant Flow.

Expectation of success

Overall, participants were optimistic about using the app, with a mean score of 12.90 (SD=2.09) out of a possible 16 points. Every single participant agreed that in principle, people could learn skills for improving sleep from an app and indicated that they felt that study participation was important. All participants reported that improving their sleep habits were important, with 49% indicating it was ‘very important’. Finally, the sample demonstrated their readiness for change with 100% of the sample indicating that they were either moderately ready (16%), ready (43%) or completely ready (41%) to improve their sleep patterns using an app.

Retention

Of the 50 participants in the study sample, 47 (94%) completed the baseline questionnaire and sleep diaries and were invited to download the Sleep Ninja. At poststudy, 34 participants completed the poststudy battery (72% retention). Participants who had available data at both time points did not differ significantly from those who only had baseline data on any of the questionnaires or sleep diary measures (all Fs <2.58, ps >0.115).

Uptake and adherence

Forty-five participants (96%) who completed the baseline assessment downloaded the Sleep Ninja. Programme usage data indicated that of these, 82% completed the first lesson, 51% completed four of the six lessons and 33% completed all six. Participants were accurate in their reporting of app use, with approximately 80% of participants indicating that they completed ‘most’ or ‘almost all’ of the app.

Acceptability

Survey responses on the Acceptability of the Intervention questionnaire indicated that young people reported that the app was ‘easy’ or ‘very easy’ to use (97%). The majority of participants (59%) indicated that they learnt ‘a fair bit’ from the app, and 28% reported that they learnt ‘a great deal’, while 12% did not learn very much or almost nothing. Participants found the app to be either ‘useful’ or ‘very useful’ (78%), while the remainder (22%) did not find it useful. Most respondents (72%) reported changing their behaviour after using the app, and examples of behavioural change included changes to their prebedtime routine (22%), keeping more consistent sleep-wake cycles (65%), getting up earlier in the morning (22%) and restricting the use of their bed for sleep (30%). More than half of the participants reported that they would use this kind of app in the future (56%), and encouragingly, 91% would recommend the Sleep Ninja app to a friend. The degree to which participants found the app useful was positively correlated with module completion, (r=0.35, p=0.047), as was the amount participants reported learning from the app (r=0.49, p=0.004).

The interview mirrored the findings of the questionnaire in terms of acceptability and usefulness. However, there were some aspects of the app that users felt could be improved. Specifically, interviewees expressed a desire for improved explanation of the different app sections and what they needed to do each time they opened the app. Participants commonly reported wanting to be able to personalise their user experience more, including skipping information they knew, seeking more information around difficult or unfamiliar topics, accessing information in different formats (eg, video/audio), being able to speed up or slow down the Sleep Ninja’s text display, and being able to update their wind-down activity choices and the time the wind-down reminder appeared. Participants expressed a range of views about the tone of the Sleep Ninja, with nearly half of the interviewed participants commenting favourably on the Sleep Ninja’s jokes, with several participants commenting that the Sleep Ninja’s language was annoying and too childish. There was consensus that the Sleep Ninja’s language was repetitive and could be improved by cutting out superfluous dialogue that was not delivering core intervention strategies. Nearly half of the interviewed participants expressed some difficulty in implementing at least one of the Sleep Ninja’s recommended strategies due to conflicting parental bedtime rules. For instance, the strategies to delay bedtime until sleepy and leave the bed/bedroom if unable to get to sleep after more than 30 min most commonly encountered parental resistance or required modification. Numerous interviewees commented on the usefulness of receiving feedback and summaries of their logged sleep, however, several commented that this could be improved by displaying the information in graphs and over time, so that change and improvement is clearer. Overall, participants reported that the number of notifications in the app was acceptable, and that additional reminders should be sent to notify them of available lessons and after periods of inactivity and that these reminders should contain motivational and encouraging messages. While most interviewees considered three nights of sleep tracking per belt acceptable, there were others who felt fewer nights of tracking would have been better, and others who expressed willingness to track more than three nights before levelling up.

Reasons for Non-Adherence

Results from this questionnaire indicated that young people were very happy to use an app to receive help for their sleep issues (84%), did not have technical issues with its use (75%) and felt they had the technical skills to use the app (90%). Participants all reported that the material was relevant and conceptually easy to understand. The main reason participants reported not using the app was that they felt it took too long to work through (53%), there was too much text to read (47%) and that it was too repetitive (59%).

Preliminary effects

Sleep and Mental Health Questionnaire Outcomes: Table 2 shows the results for the questionnaire measures. As predicted, from baseline to poststudy, there was a significant decrease in insomnia severity measured by the ISI, β=−4.29, p<0.001, d=−0.90, and sleep quality on the PSQI, β=−1.88, p<0.001; d=−0.46. Similarly, mental health measures showed a decrease in both depression on the PHQ-A, β=−2.60, p<0.001, d=−0.36 and anxiety on the GAD-7, β=−2.56, p<0.001, d=−0.41.

Sleep diary outcomes: Results for the sleep diary entries are shown in table 3. At baseline, participants went to sleep, on average at 23:29 hours, reported taking an average of 1 hour and 12 min to fall asleep, spent an average of 9 hours and 39 min in bed, woke up an average of 1.47 times, slept for a total of 7 hours and 40 min, woke up at 7:27 hours, and spent approximately 29 min awake in bed before getting up. Overall SE was just above 80%. Results from the analysis at postintervention indicated that as predicted, participants went to bed 35 min earlier then at baseline (β=−0.58, p=0.003), there was a significant decrease of 21 min in how long participants took to fall asleep (SOL; β=−0.37, p=0.032), participants spent significantly less TIB after waking than they did at baseline (β=−0.27, p<0.001), and woke significantly less frequently during the night reducing to an average of 0.87 times (NWAK; β=−0.46, p=0.011). There were also improvements in TST of 33 min (TST; β=0.53, p=0.005), SE (β=5.25, p=0.016), how refreshing sleep was (β=0.43, p<0.001) and sleep quality (β=0.31, p=0.018). There were no significant differences in the time participants woke up in the morning (on average, at 7:20 hours), TIB, WASO or medication use (all ps>0.05). Within-group Cohen’s d effect sizes ranged from small to medium (0.31–0.68).