Objectives Inspired by the James Lind Alliance (JLA) user involvement approach, the aim of the present study was to identify the top 10 uncertainties for sleep research raised by students in higher education, and to discuss our experiences with adapting the JLA method to a student population.
Design The study design is a pragmatic JLA approach, including a priority setting partnership within the field of sleep, collection of sleep-related research uncertainties as reported by students in higher education, sorting of the uncertainties and a final identification of the top 10 uncertainties through collaborative work between researchers, students, stakeholders and experts in the field. Uncertainties were collected using a one-question online survey: ‘as a student, which question(s) do you consider to be important with regards to sleep?’. A variety of approaches were applied to promote the survey to the students, including social media, radio, the university website, stands in university cafeterias and a sleep stunt. NVivo V.12 was used to code and sort the questions.
Setting A higher education institution in Norway.
Participants 555 students.
Results The data collection provided 608 uncertainties, and the following prioritised top 10: (1) screen time, (2) stress, (3) educational achievements, (4) social relations, (5) mental health, (6) physical activity, (7) indoor environment, (8) substance abuse, (9) shift work and (10) sleep quality. Despite successful data collection, we found sleep to be a broad topic, and defining specific questions throughout the sorting and verification process proved difficult.
Conclusions We identified the prioritised top 10 research uncertainties as reported by students in higher education, ranking screen time first. However, the process was time and resource consuming. The research uncertainties addressed by the students showed great diversity, characterised by heterogeneity and a lack of specificity, making verification of the uncertainties challenging.
- public health
- sleep medicine
- statistics & research methods
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
Students have not previously been invited to participate in research concerning what they consider important regarding sleep and which topics that are relevant in sleep research.
We applied a pragmatic James Lind Alliance approach and provided a list of research uncertainties for future studies on students and sleep.
The student respondents were recruited from a university offering primarily studies in applied sciences, which might not represent all students at university level.
We abstracted the uncertainties to facilitate efficient verification; however, the nuances of some questions could have been lost in the process as we abstracted the uncertainties, first into questions and then into topics.
There has recently been an increased focus on the discrepancy between what users want researchers to study and what researchers actually investigate, as well as a greater attention to reduce research that is asking the irrelevant questions, using wrong designs, preparing weak reports or even biased reporting, often labelled as research waste.1 2 The importance of user-involvement and needs-identified research has been highlighted in Norway3 as well as internationally.4 Nevertheless, the extent of user involvement in research varies, mostly due to practicalities, since user involvement can be both costly and time consuming.5 6 It is increasingly recognised that knowledge is not merely produced by researchers.7 User involvement to identify relevant research topics and research questions can increase research quality, ease recruitment and participation, reduce participant dropouts,6 8 and ultimately ensure knowledge development and research that is relevant to the public. One method of involving users is the James Lind Alliance (JLA) method,9 which aims to ensure research that answers questions that users, peers or relatives, and stakeholders agree reflect the most important issues. The JLA approach entails a stepwise process of establishing a group of researchers and representatives from the group of interest and together design a study to gather the research uncertainties within a group.9 Furthermore, the approach consists of steps to sort and prioritise these uncertainties until a list of top 10 research uncertainties is complete. JLA has received great attention within numerous areas of the health research field, targeting patients, their relatives, healthcare personnel and relevant stakeholders to address treatment uncertainties. Previously, the JLA method has mainly been applied to study treatment uncertainties in specific diagnosis-related populations, while it has rarely been used to study health issues in the general population, although examples of this do exist.10
The public’s involvement in sleep research remains scarce, and students as a specific target group have not previously been approached to gather data on uncertainties, despite sleep problems being highly prevalent in this population.11 12 Asking students in higher education about their research uncertainties concerning sleep can thus provide valuable insights for future sleep research topics carried out within this population.
Sleep problems have been found to have a multitude of negative consequences,13 14 where lack of sleep is frequently addressed as a public health crisis.15 Sleep problems and sleep loss can affect daytime functioning and are associated with impaired cognitive functioning,16 which has been attested in a wide range of cognitive tasks.17–27 Evidence also suggests that motor performance is significantly impaired by poor sleep.28 29 Other negative health effects associated with sleep problems comprise increased risk of obesity, Alzheimer’s disease30 or depression,31 and an overall decrease in health-related quality of life.32 As such, sleep problems have increased costs and economic societal consequences, and threaten public health.33 34 Although all humans can experience sleep problems on occasion and it has been described since ancient times, they are on the increase and insomnia, characterised by difficulty falling asleep, disrupted sleep or lack of restorative sleep, is now present in between 6% and 15.5% of the adult population.35 36
In the recent Student Health and Well-being Survey carried out among Norwegian full-time students in higher education aged <35 years, one in three students reported symptoms of deteriorated sleep to such degree that it meets the requirements of an insomnia diagnosis with difficulty falling asleep, disrupted sleep or lack of restorative sleep.12 37 Previous research suggests that sleep problems negatively influence participation in school, daily activities and working life. Several factors make students in higher education especially vulnerable in terms of sleep problems.12 38 First, students are in a period of transition. Many have moved away from home for the first time and live on their own. They aim to balance educational demands, make new relationships and friends, navigate in a new place and maintain a healthy lifestyle with good sleep hygiene practices. Increased use of digital media, high caffeine intake among students,39 lack of daily structure and use of alcohol and other recreational drugs40 can also threaten their normal sleep, which in turn might be associated with the absence or dropout from higher education.39 Furthermore, poor sleep is associated with lower socioeconomic status among young adolescents.41 Thus, the population’s sleep health is a public health concern,42 especially among students, since it is essential to assure their participation in education and later working life.
Inspired by the JLA methodology for user involvement, our aim was to identify the top 10 uncertainties in sleep research communicated by students in higher education, and to discuss our experiences with adapting this JLA framework to a public health setting. This novel approach will provide insight for future research in students and sleep framed by the students, and we will share our experiences with the pragmatic JLA approach allowing for user involvement.
Patient and public involvement
The JLA methodology in which the current study is inspired by is founded on the principles of patient and public involvement. As such, users have been involved in the entire process of this project, from study design to the results of the top 10 list. Our users were mainly students in higher education, but also stakeholders. The information regarding their involvement will be described in detail in the following sections.
The current study was inspired by the JLA methodology, which aims to address research questions that are relevant for a specified group.9 In JLA, these unanswered research questions are referred to as uncertainties, and can be formed as questions, themes or memos containing something on which the target group is asked to reflect, before the uncertainties are ideally reframed into researchable questions. We interpret the framing of the uncertainties as more open themes, allowing greater flexibility in terms of the research methods that can be used to investigate them. In order to gather, sort and prioritise the uncertainties, the involved partners must work systematically together according to the priority setting partnership, as outlined in figure 1 and presented consecutively.
Establishing the priority setting partnership
This first step of the priority setting aims to address the scope of the project, invite members and develop a project protocol with the members of the partnership.9 The quality of life research group at the Faculty of Health Sciences at Oslo Metropolitan University (OsloMet) initiated the current project in 2018. Through the initial phase, a steering group was established consisting of researchers and relevant research collaborators, student representatives, members of the student parliament and faculty leaders at different levels, all situated at OsloMet. A 2-hour workshop was held, with a view to carefully plan the priority setting process. Attendees at the workshop were researchers and a representative from the Faculty of Health Sciences, two researchers from the Faculty of Technology, Art and Design, and three undergraduate students from the nursing programme. During this workshop, we planned and discussed how to recruit a convenience sample of students in higher education, represented by students in all ages at OsloMet. The insight from the students who participated at the workshop was given much emphasis when planning how to gather the uncertainties, in order to tailor our recruitment and data collection strategy to both social media and campus on-site. A strategic project plan was developed, addressing the required permissions, recruitment strategies, and choice of data collection method and solution.
Gathering the uncertainties
According to JLA, this phase aims to collect the uncertainties in line with an a priori project plan.9 To ensure appropriate recruitment strategies and data collection procedures, the survey for gathering uncertainties was pilot tested by a small sample of students. Second-year nursing students developed different questions for collecting data and tested these on a group of their peers through focus groups and anonymous questionnaires. The students presented the results from this pilot to the researchers. The students and researchers consequently agreed on a questionnaire, appropriate recruitment strategies and a data collection procedure.
In collaboration with the Department of External Relations and Communications at OsloMet, we designed a week-long sleep event aiming to ‘tune in’ the students towards sleep and sleep health, culminating with the World Sleep Day, on 15 March 2019. The sleep event was promoted online through Facebook, Instagram and Snapchat, and on the university’s home page to ensure the students’ awareness of and interest in the sleep week and prepare them for the data collection (figure 2). All platforms provided a link to a Norwegian online survey that could be accessed using PC, Mac, tablet or smartphone. Campus visibility was facilitated through information monitors, providing facts about sleep and the sleep week event. We also hosted a sleep stunt, where two students were chosen to sleep out in public for two nights in a campus cafeteria, where they were filmed and interviewed by members of the research group before they went to bed and the following morning. The event was published on various social media platforms, all aiming to call attention to student sleep and the students’ opportunity to submit their uncertainties. We also had researchers available at lectures and at the campus cafeterias during lunch hours for 5 consecutive days. Here, pairs of researchers interacted with students, emphasising the students’ chance to set the agenda for future research on sleep through the online survey. In addition, a radio show had two interviews with the two students who took part in the sleep stunt, two researchers and the vice rector for education. Lastly, we wrote an op-ed highlighting the need for sleep research carried out in the student population.
Uncertainties were collected by asking one question: ‘As a student, which question(s) do you consider to be important with regards to sleep?’. The uncertainties were gathered from students at OsloMet, and data collection was completed through a digital, anonymous survey, where no other data or IP addresses were stored.43 In this way, it was not possible to identify the participants, and the Norwegian Centre for Research Data approved the data collection strategy. A priori, we aimed for approximately 500 uncertainties based on pragmatic consideration of our resources. The survey asked the participants to confirm that they were students and provided a link to a website for further project information. The students’ submission of their uncertainties was recognised as their consent to participate.
Verifying the uncertainties
Following data collection, the JLA advises verification of the uncertainties, aiming to make sure all the data collected are in fact research uncertainties, and that they have not been satisfactorily answered by previous research.9 All data were imported into Microsoft Excel to sort the text lines, incorporating the uncertainties, before the file was split into four and entered into NVivo. Four groups of five or six researchers verified their dedicated file of uncertainties by removing non-students, duplicates, reframed topics and memos, and ensured that all uncertainties were within the intended scope of sleep. Altogether, there were N=558 entries (see figure 3). Of these, we excluded three entries that were made by non-students.
Some entries contained several questions. One example is ‘What is the best sleeping position, how many hours of sleep do I need, and is it important to take a nap during the day?’, an entry containing three uncertainties. Thus, n=555 entries provided at least one research question/uncertainty/theme, and when we split all entries into single uncertainty, there were 608 uncertainties. These were sorted and abstracted by each group (see table 1) before all lists were merged, duplicates between groups removed and the groups met to discuss the compilation of a first long list comprising n=119 uncertainties. After removing overlapping questions, the collapsed long list contained n=48 abstracted questions. Table 1 illustrates how the initial uncertainties were abstracted into topics, through a selection of the abstractions with the richest data.
According to the JLA methodology, the initial long list must be verified in relation to previous research.9 The nature and number of the uncertainties collected about sleep in a general student population did not allow us to address specific literature searches to verify previous evidence for each of the uncertainties, as the JLA suggests. Instead, we were inspired to pursue an alternative approach. To establish the current evidence base to answer the question: ‘What is known about students and sleep?’, we initiated an evidence map where we systematically searched, screened, classified and depicted all research on students and sleep published after the year 2000 (Blinded). This evidence map was initiated to act as an evidence bank for the overall project and will be published in a separate paper. As such, previous evidence did not have any consequences for the list of uncertainties in the current project; that is, none of the uncertainties were removed from the list at this stage.
Interim priority setting
The aim of the interim priority setting is to sort and prepare the long list and reduce the list to a short list.9 Thus, the long list of uncertainties was presented and thoroughly discussed among the researchers. A designated group of six researchers independently abstracted the long list before they met to compare their lists and finalise a preliminary short list of uncertainties. This list of 48 uncertainties was further discussed in the research group on several occasions as a form of face validity for the previous steps. In accordance with the evidence map suggesting a need for research with various scientific methods, and as a method of ensuring a similar level of specificity in the uncertainties, all questions were thematically reframed as topics, providing us with a short list of n=25 abstracted topics (table 1).
Final priority setting
The final priority setting process aimed to finalise a list of top 10 uncertainties.9 We hosted a workshop and used a participant observation technique in order to evaluate the process,44 and documented observations through notes and photos. All participants were made aware of the documentation of the workshop beforehand, and that notes and pictures would be taken. In addition, a sample of researchers were invited to submit their immediate impressions from their group discussions. We used secondary analysis to interpret the observations and participant quotes44 with the aim of learning about the realities of the group’s experience and behaviours.45
At the final workshop, 20 researchers, 7 students, 2 leading experts within the field of sleep medicine and research, and 3 stakeholders participated. All participants were divided between six groups, with students, stakeholders and experts represented in most of the groups, consisting of between two and six participants (median five). To begin with, the aim of the workshop was presented, and the participants read the short list and shared their understanding and discernment with the group through a 40-minute discussion, prior to voting towards the top 10 list. All groups experienced lively discussion, allowing every participant to state their opinion, including the students. It was random among the groups which group member led the discussions. In one group, a student took on this role, while a researcher was the leader in two groups and a stakeholder in another. Student engagement varied among the groups from being actively involved in the discussion to only speaking when encouraged by the other group members. There was no conflict related to the possible meaning of the topics or their relevance for future research, and there seemed to be an unspoken agreement that allowed for different opinions, as the final voting was to be individual.
The interactive web-poll Menti by Mentimeter (www.menti.com) was used to vote for the top 10 uncertainties. Participants individually chose the 10 most important topics without ranking them, as the digital tool Menti did not allow for individual ranking. Thus, our top 10 is based on the number of votes cast by the participants as a group. This voting provided an immediate presentation on the screen. The topics receiving the same number of votes were included in a new poll to decide their internal ranking.
Participation was voluntary and no data other than the students’ proposed research uncertainties were collected. Submitting the uncertainty in the online survey was recognised as informed consent. Complementary information about the study was available on the study website. In accordance with the institution’s guidelines, a risk and value assessment was performed a priori with no risks identified. The institution’s local data protection officer was consulted to ensure that privacy regulations were followed.
The two students involved in the sleep stunt were carefully recruited and informed of the possible consequences of sleeping in public, and they were able to contribute to the stunt’s development. Two designated researchers were available for questions and to address reactions that could arise during the stunt.
The top 10
The short list presented to the participants contained the following themes within the context of students and sleep: being responsible for young children, dreams, physiological processes, physical activity, indoor environment, caffeine, dietary habits, chronotypes, power naps, mental health, substance abuse, sex, screen time, social relations, sleeping position, sleep quality, sleep duration, stress, educational achievements, disease, sleep measures, gaming, tiredness, shift work and external environment. All participants at the final workshop (N=30) were instructed to choose 10 of these themes without ranking them (table 2).
After the first round of voting, several themes received the exact same number of votes. The result was a shared second place between two topics (educational achievements and stress), and a shared sixth place between four themes (indoor environment, shift work, sleep quality and substance abuse). To solve this, and thus create a ranked list of unique placings, we immediately conducted a second round of voting. The participants first voted between educational achievements and stress, where stress received a higher number of votes, and was consequently placed in second place in the ranking. The participants then voted between indoor environment, shift work, sleep quality and substance abuse. Table 3 shows the final top 10 list of topics ranked by number of votes.
Our process was initiated as a joint project involving some 20 researchers. We started in June 2018, and carefully planned the project until the online survey opened on 10 March 2019, marking the start of sleep week and the data collection period. Data collection consisted of pairs of researchers engaging students for an hour each day at three locations for 5 consecutive days, in addition to being available at lectures. Several researchers also contributed their time during the sleep week in the campus area. All of these activities were crucial to the data collection process, and ensured collection of a high number of uncertainties.
The students submitted a wide range of uncertainties. The majority of submitted uncertainties contained the word ‘affect’, and were concerned with the effect of various interventions, for example, ‘What is the effect of drinking coffee in the evening?’ and ‘What is the effect of using a mobile phone to watch a film immediately before falling asleep?’ (see table 1 for more examples). We found that effect studies are not prevalent in the sleep research field, which is more dominated by cross-sectional studies. Previous research did not lead to the exclusion of any of our uncertainties. In the verification of the uncertainties, we abstracted the uncertainties to make them manageable prior to the voting.
To the best of our knowledge, this is the first JLA study to approach students, and the first presentation of uncertainties on students and sleep gathered from students in higher education. The JLA process is comprehensive and resource-intensive in order to fulfil its intentions as a systematic approach to identify research questions framed by the users. We argue that this is not a process individual researchers can successfully conduct on their own. Thus, the current list of top 10 topics is the result of collaborative and pragmatic work between researchers, students and stakeholders.
Although our group consisted of several dedicated researchers, we experienced particular challenges with the verification of uncertainties. Adapting the JLA process to public health, rather than addressing treatment uncertainties in specific diagnosis-related populations, was also experienced as demanding. Identifying sleep problems can be more challenging than, for example, use of drugs or smoking, due to the close ties to other disciplines, and because of the emotions often involved in sleep as a topic. We chose to investigate the topic ‘students and sleep’ because of the alarming increase in sleep problems among students,11 12 and the detrimental consequences lack of sleep can have for students facing these challenges.38–41 Our top 10 list addresses a variety of themes within the selected topic, covering many aspects of a student’s life. Interestingly, in the final ranking, ‘sleep quality’ was ranked the lowest. This was somewhat surprising as sleep quality can have consequences for all the other themes. However, sleep quality will expectedly be included and addressed within the other topics. The ranked list reflects current debate on student life, being a balancing act between screen time, educational achievements, stress, social relations and mental health. All of these topics appear in the media on a weekly basis, and we were not, therefore, surprised that they were ranked as important. It appears that student life represents conflicting activities and a higher pressure to achieve good results than previously,46 as also reflected in our top 10 list.
The current gathering of uncertainties had some deviations from the process outlined by the JLA. Compared with specific diagnosis-related populations, students can comprise a more diverse and heterogeneous group of people in terms of demographics, previous medical history and sleep habits, which had consequences for the unanswered questions they submitted and our sorting of these later.
The majority of the uncertainties contained the verb ‘affect’, indicating that students are concerned with how sleep affects their lives on the one hand, and which factors in their lives affect sleep on the other. Thus, the majority of the students’ concerns would entail causal or explanatory research designs in order to provide answers. This calls for interventions to optimise the interaction between student life and sleep. When we first initiated systematic literature searchers to verify the unanswered questions, we realised that the field lacked experimental research that was needed to verify the questions as true uncertainties according to JLA. Thus, our literature review on students and sleep resulted in an evidence map. During the verification step, the uncertainties raised by the students were abstracted to successfully integrate and collate them into manageable uncertainties, similar to an approach used in qualitative content analysis.47 Abstracting the uncertainties facilitated efficient verification. However, the nuances of some questions could have been lost in the process as we abstracted the uncertainties, first into questions and then topics. Nonetheless, we consider the open framing of our top 10 as topics rather than questions to be a strength that allows future research to investigate these knowledge gaps using suitable and appropriate scientific methods. A rigid framing of 10 research questions would demand a specific scientific method to be applied in order to answer these in future research. Complimented by the evidence map conducted by our group, the current top 10 topics allow future research to investigate different aspects of the topics ranging from effect to personal experiences.
Our sample comprises students in higher education from one single university in Norway. The university has four faculties from which we aimed to collect data, namely, health sciences, education and international studies, social sciences, and technology, art and design. As the purpose of the JLA process is to gather research uncertainties from a target group and the relevant stakeholders associated with this group, we do not regard our lack of a strategically drawn and representative group to be a limitation in this context. As this group of students comprised a general sample with varying interest in sleep, we needed to draw some attention towards sleep in order to set the agenda for the data collection, but without giving the student specific ideas as to what they should submit in terms of questions. Still, it could be argued that this may have left a biased impact on our data collection. The range and variability of the submitted uncertainties, however, suggest that we succeeded with our neutral awareness raising. The initial uncertainties were highly overlapping, suggesting that we achieved saturation for this specific target group. If we had only included students with a previous history of sleep problems, it is very likely that the uncertainties would have been more detailed or specific in their wording, and had possibly been easier to sort and verify, but with a much narrower focus (eg, on pathology). This would have had less value for future public health research.
The top 10 list of research uncertainties relating to students and sleep is available to all researchers interested in pursuing new research projects. Together, our top 10 list of uncertainties and the results of the evidence map will contribute unique knowledge that can inform future research. Researchers can choose the topics of relevance from the top 10 list, use the evidence map to investigate particular evidence gaps and the methodological approaches appropriate for future research, and design new projects accordingly. Seen as a whole, this will increase the relevance of future research, and reduce the amount of research waste.
We applied a pragmatic approach that is more suitable for a student population, including abstraction of the submitted research uncertainties and thematic presentation of the top 10 as topics through: (1) screen time, (2) stress, (3) educational achievements, (4) social relations, (5) mental health, (6) physical activity, (7) indoor environment, (8) substance abuse, (9) shift work and (10) sleep quality. Reporting this process and the top 10 evidence uncertainties about students and sleep provide valuable insight for the research society, which will help to set the agenda for relevant future research and reduce research waste. However, we experienced the JLA process to be resource demanding and complex, and as a result, JLA may not be feasible for researchers who do not have a large research group to lean on.
The authors wish to thank all the students who contributed with their research uncertainties. A special thank you to the students in the steering group, and those who pilot-tested the data collection strategy. We wish to thank the two associate research members, Liv Klakegg Dahlin and Sara Birgitte Øfsti Nesje, at the Faculty of Technology, Art and Design at Oslo Metropolitan University (OsloMet) for initial contribution in this process. We are very grateful for the support from the Department of External Relations and Communications at OsloMet. In addition, we are grateful for the important contributions made from the stakeholders throughout this process, and the contributions made from the leading experts at the final workshop.
Contributors The quality of life research group at Oslo Metropolitan University initiated and designed the project, and completed the priority setting process. HH and EKG were the principal investigators and provided conceptual guidance for the remaining research team and the involved students. HH, KR, BL, IHS, EKG, KA, RA, AKB, EAF, SH, LGK, NM, AR, IR, LR, HMR, BS-N, AT, IU, LV and AW participated in the first seminar in 2018 to design the study in detail. HH, EKG, AKB, SH, LGK, SP, LR, BS-N and BTV drafted the project protocol and sought necessary approvals from the department and data protection officer. HH conducted the risk and vulnerability analysis. BL, KA, EAF, ML, NM, IR and IHS designed the marketing campaign to secure recruitment. BL and IHS specifically led the sleep stunt. KR, RA, BHF, AR, HMR, AT, IU, LV and AW led the process to initiate the digital data collection. HH, KR, BL, IHS, KA, RA, AKB, BHF, EAF, SH, LGK, ML, NM, AR, IR, LR, HMR, BS-N, AT, IU, BTV, LV, AW and EKG participated in the data collection. HH, KR, BL, IHS, KA, RA, AKB, BHF, EAF, SH, LGK, ML, NM, AR, IR, LR, HMR, BS-N, TMS, AT, IU, BTV, LV, AW and EKG analysed data through either meetings, workshop or seminar. HH, KR and EKG drafted the manuscript, and all co-authors commented on and revised the manuscript. All authors have read and approved the manuscript.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient consent for publication Not required.
Ethics approval The current project was presented to the Data Protection Officer at the independent Institutional Review Board at OsloMet with no comments and no need for further ethical approval.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data are available upon reasonable request. The datasets generated and/or analysed during the current study are not publicly available but are available as raw data in their original language, Norwegian, from the corresponding author upon reasonable request.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.