Article Text
PDF
Abstract
Introduction Massive Open Online Courses (MOOCs) are informal learning environments. Since a few years, MOOCs are being reused and integrated in formal medical education. However, what constitutes optimal integration is still unclear. In this mixed methods study protocol we describe how we will investigate three MOOC integration designs using the same MOOC.
This study holds multiple objectives (1) Describe motivation profiles in medical students that learn in integrated MOOCs, and discern if motivation profiles are associated with specific MOOC integration designs; (2) investigate how psychological needs of medical students are satisfied or frustrated in different MOOC integration designs; (3) investigate the relationship between autonomous motivation to learn in an integrated MOOC and use of self-regulated learning skills in that MOOC; (4) uncover processes that are involved in goal acceptance or rejection of medical students in integrated medical MOOC designs with assigned learning goals; and (5) identify obstacles medical students encounter when learning with assigned learning goals in integrated medical MOOCs.
Methods and analysis Objectives 1 and 2 will be pursued with a cross-sectional study design, objective 3 with an observational cohort study design and objectives 4 and 5 with a qualitative interview study design. All medical students in one of three MOOC integration designs at Leiden University Medical Center (LUMC) will be invited to participate. Primary endpoints for objectives 1 and 2 are motivation profiles, and variety in need satisfaction and frustration. For objective 3 the primary endpoints are autonomous motivation and self-regulated online learning. For objectives 4 and 5 primary endpoints are process themes regarding goal acceptance or rejection, and perceived obstacles when working with assigned online learning goals.
Ethics and dissemination This study has been approved by the Educational Research Review Board of the LUMC. Planned dissemination of findings include three presentations at (inter)national conferences and three research articles.
- medical education & training
- world wide web technology
- education & training (see medical education & training)
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
- medical education & training
- world wide web technology
- education & training (see medical education & training)
Strengths and limitations of this study
The used Massive Open Online Course (MOOC) is open for integration in other institutions, the teaching modes profile will be disclosed and integration designs are described extensively, increasing replicability to a high extent.
By comparing integration designs using the same MOOC, we maximise the validity of the findings regarding differences between integration designs.
This study uses three data collection points in time for each participant to answer a total of five research questions, minimising the participants’ time investment.
Specific efforts have been made and described to optimise the quality of both quantitative and qualitative methods.
As only one MOOC is investigated, future research has to decide on generalisability to integration of MOOCs with different teaching mode profiles.
Introduction and rationale
Massive Open Online Courses (MOOCs) are informal learning environments that are mostly created by universities. In MOOCs, learners from all over the world are free to choose any topic, at any place and any time to learn, usually with little or no financial commitment needed. In addition to this traditional MOOC format, many other MOOC forms now exist, ranging on scale, openness and costs for learning.1 Since a few years, MOOCs are being integrated in formal campus education,2 with many examples of integration in the medical domain because of the advantage integration offers: (1) the convenience of creating a course once and delivering it multiple times without extra effort or cost,3 (2) access to education from institutions that not all students can travel to,4 (3) the opportunity to remove costs and inconvenience of getting to a single location,5 (4) access to topics not normally available in the curriculum,4 (5) the possibility to use ‘exemplar’ learning materials from experts in their field instead of each university making their own,4 6 (6) enhanced understanding of pathology not common to students’ resident country,6 (7) enhanced communication among international communities of clinicians and student clinicians,7 8 (8) access to a wide variety in available teaching modes7 and (9) access to innovative teaching models for student learning.7 8
Many studies have described the way a MOOC was integrated into the campus context,9 sometimes accompanied by outcome measures such as student satisfaction,10–12 or effectiveness for learning,13 and in 2019, an article describing 12 tips for integrating medical MOOCs into campus education was published, based on the experiences of early adopters and researchers of MOOC integration.14 However, what constitutes optimal integration is still largely unclear, as most studies only describe one case of integration while integration contexts differ significantly between cases. MOOC integration designs can be characterised by choices of (1) level of education, (2) degree of obligation, (3) ratio of online versus face-to-face teaching, (4) replacing or adding MOOC content to formal courses and (5) level of contact with other online learners in the MOOC. A MOOC integration design is thus a combination of choices in each of these five areas. We see this distinction between designs as a first step towards practical insights into what works when and why. In this proposal, we present a mixed methods study that investigates three MOOC integration designs using the same MOOC, and explore motivation to learn and self-regulated learning (SRL) skills in this context. We have outlined our three directions for research below, which are all exploratory and descriptive, and function to form hypotheses for future research. In short, we seek to discern;
How medical students are motivated to learn in three different MOOC integration designs,
If autonomous motivation is needed for self-regulation when learning in an integrated MOOC, and
What processes and obstacles are involved in working with assigned learning goals while learning in an integrated MOOC setting.
These directions give form to a total of five research questions, of which the rationales will be described below.
(1) Theoretical framework for RQ1 and RQ2: motivation in different MOOC integration designs
Self-determination theory distinguishes between quantity of motivation and quality of motivation.15 One can be highly motivated, but when this motivation is only externally regulated, or controlled, it is considered low quality motivation.16 High quality, autonomous motivation is more internally regulated, and is associated with well-being, enjoyment and academic achievement.15 17 Self-determination theory also postulates that in order to be autonomously motivated, there is a psychological need for feelings of autonomy, feelings of competence and feelings of relatedness to others. In educational settings, these feelings can be satisfied or frustrated, which satisfies or frustrates autonomous motivation, which in turn influences the quality of motivation.17 18 Motivation to learn in integrated MOOC settings is a relevant and understudied outcome measure. To gain insight in the quality of motivation in medical students in integrated MOOCs, we aim to answer the following research questions:
RQ1: What are motivation profiles of medical students in three different MOOC integration designs, and do the three integrated MOOC designs differ in students' motivation profiles?
RQ2: How are psychological needs of medical students satisfied or frustrated in different MOOC integration designs?
(2) Theoretical framework for RQ3: autonomous motivation and use of self-regulated learning skills
In addition to the benefits of well-being, enjoyment and academic achievement, autonomous motivation is thought to stimulate SRL.17 It is widely accepted that online learning demands more SRL strategies, as usually no teacher, tutor or mentor is present.19 Many studies have focussed on what processes are involved in SRL, and subsequently strategies were developed to teach successful execution of SRL. Recent literature reviews suggest that SRL processes can be supported by adding SRL prompts, feedback or a combination of the two for many of the SRL processes, including goal-setting, monitoring and evaluating. However, although SRL strategies may be successfully acquired, even online, this might not be enough to make students actually self-regulate their learning when it is no longer supported. In the book Motivation and Self-Regulated Learning by Schunk and Zimmerman (2008), Reeve et al state:
’We believe that developing such skills is important for students’ regulating their learning activities effectively. However, we also suggest that for students to put the skills to use and take greater responsibility for their learning, they will need to develop autonomous motivation for doing so.’
- Reeve, Ryan, Deci and Jang, page 239
The authors suggest a two-tier condition for students to self-regulate: they must know how to, and they must want to do it for themselves. Recent literature suggests that efforts to support SRL in MOOCs focus on offering support for how to self-regulate, and not on autonomous motivation for doing so. If Reeve et al are right, efforts will also have to be directed at supporting autonomous motivation in integrated MOOC designs. As we have found no studies to test this relationship in online learning settings we seek to investigate their assumption. Findings can inform research efforts to support SRL in MOOCs, and can offer guidelines for future MOOC integration designs. The related research question is:
RQ3: What is the relationship between autonomous motivation to learn in an integrated MOOC and self-regulated learning in that MOOC?
(3) Theoretical framework for RQ4 and RQ5: goal setting processes surrounding assigned online learning goals
Goal setting is an important part of SRL and it has been described as an essential skill for learning in MOOCs.19 Students that set their own learning goals are more autonomously motivated, set more difficult goals, show higher commitment and greater affect when attaining or not attaining a goal.20 When possible, self-set goals are to be preferred over assigned goals.20 However, in most medical MOOCs, like in most other courses, learning goals are still assigned. As we strive to have more self-regulated students, this goal assignment might pose a problem. A possible solution for this problem is to set goals with student and teacher together, as is posed by goal-setting theory21 and the social-cognitive path to self-regulatory skills.22 However this requires individual attention of the teacher for each student. This might be hard to achieve in MOOCs, with few teachers and many learners, and with predetermined learning activities and assessments that might not fit the new learning goals. Latham, Erez and Locke (1988)23 have suggested acceptance of a goal is more important than its’ origin, that is, the student or the teacher. In this line of thought, student acceptance and internalisation of assigned learning goals might offer the solution that is needed for autonomous motivation to learn. Difficulties with assigned learning goals and co-creating learning goals have been described in multiple studies in clinical learning contexts,24 25 but we have not come across literature that describes learning goal acceptance in online learning settings. For this reason we seek to gain insight in the processes that are involved around goal acceptance of medical students in integrated MOOC designs with assigned learning goals. Related research questions are:
RQ4: What processes are involved in goal acceptance or rejection of medical students in integrated MOOC designs with assigned learning goals?
RQ5: What difficulties do students perceive in working with the assigned goals, and what helps them when working with assigned goals?
Objectives
To summarise, in our study the following objectives are pursued:
Establish motivation profiles of medical students in integrated MOOCs, and discern if motivation profiles are associated with specific MOOC integration designs.
Determine how psychological needs of medical students are satisfied or frustrated in different MOOC integration designs.
Identify the relationship between autonomous motivation to learn in an integrated MOOC and self-regulated learning skills in that MOOC.
Uncover processes that are involved in goal acceptance or rejection of medical students in integrated medical MOOC designs with assigned learning goals.
Identify obstacles and promoting factors that medical students encounter when learning with assigned learning goals in integrated medical MOOCs.
Study design
Context description
Students in three medical MOOC integration designs will be invited to participate in our study. In each of the integration designs the MOOC ‘Clinical 'Kidney, Pancreas and Islet Transplantation’ was integrated in undergraduate courses at Leiden University Medical Center (LUMC) in the Netherlands. The LUMC is a public academic hospital in a highly urbanised region. About 320 students start their bachelor studies in the faculty of medicine each year. An overview of the characteristics of the MOOC can be found in figure 1.
MOOC characteristics. Approx., approximately; MOOC, Massive Open Online Course.
Integration design A consists of completing the MOOC before joining the 3.5-day undergraduate level ‘Leiden Oxford Transplantation Summer School’ (LOTS) which runs yearly in July.26(Leiden University website 2019a) Joining this course is voluntary and acceptance of students is based on a letter of application. However, once accepted into the course, completing the MOOC is a prerequisite to come to the face-to-face meeting. Since 2017 the MOOC has been added to equalise and enhance the level of prior knowledge among students. Students do not meet before the face-to-face component and have to enrol in the MOOC themselves, where they will learn alongside all online MOOC learners. Approximately 20 students join this course and we strive to include two cohorts of this course in the study.
Integration design B is an 8-week compulsory second-year module called ‘Mechanisms of Disease’ starting in October in which 1 week of lectures at the end of the course has been replaced by a part of the activities in the MOOC. In this design, the entire cohort of approximately 300 students has followed undergraduate courses together for a little over a year. Students are enrolled in a separated version of the MOOC course and thus have no contact with MOOC learners outside of their cohort.
Integration design C is a newly offered elective for undergraduate students that have enrolled in the honours programme, and students from universities in the virtual exchange programme.27 28 The honours programme is available for students that long for more challenge in their studies. To complete the honours programme, students must gather 30 extracurricular study credits. Students can choose the components in their honours programme from several lectures, meetings and courses, among which the MOOC. For all students in this integration design the MOOC elective consists of completing the MOOC at any time in their first or second year of undergraduate studies and an additional written assignment. Students do not meet face-to-face with other students as it is an individual online course. Approximately 14 to 18 students enrol during a study year, which is the period we will include students in. Characteristics of integration designs A, B and C have been summarised in figure 2, according to the possible integration design choices described above.
MOOC integration designs, design choices, course planning and data collection planning during the study. In design A students can decide when to complete the MOOC before the face-to-face component in July. In design B students enter the MOOC in October as part of an 8-week course. Design C is continuously available. F2F, face-to-face activities; Hons, Honours programme; LOTS, LeidenOxford Transplantation Summer School; MOD, Mechanisms of Disease; MOOC, Integrated parts of the Massive Open Online Course; T1, time point 1; T2, time point 2.
Research design
Motivation in different MOOC integration designs
For this cross-sectional study, the variety of motivation quality profiles over MOOC integration designs will be calculated to answer RQ1. To answer RQ2, scores for psychological need satisfaction and frustration will be compared between MOOC integration designs.
Autonomous motivation and use of self-regulated learning skills
To answer RQ3, we will use a prospective observational cohort study design. All participants will be handled as one group and autonomous motivation and SRL data will be used in a cross-lagged panel design, to examine the correlations between the levels of autonomous motivation and SRL scores at two points in time: at the start and at the end of the MOOC.
Goal setting processes surrounding assigned online learning goals
RQ4 is exploratory and a qualitative research design is applied. We want to understand the processes involved in assigned learning goal acceptance or rejection in integrated MOOC learning from the perspective of the students through interviews. Although some research has pried into goal acceptance, and self-determination theory could inform discussion about internalisation of assigned goals, to our knowledge, no theory is known regarding this subject and so we opt for a grounded theory approach analysis. For RQ5 we seek to map obstacles and difficulties that students encounter while learning with assigned goals in integrated MOOC. We deem Cultural Historical Activity Theory, an appropriate lens29 to interpret the wide variety of difficulties and promoting factors that can arise in such a complex learning setting.
Study population
Population
Medical students that have enrolled in one of three described MOOC integration designs at the LUMC will be invited to participate. This includes students from other universities that have enrolled as exchange students. Students will approximately be between 18 and 23 years of age.
Inclusion and exclusion criteria
In order to be eligible to participate in this study, a subject must meet all of the following criteria. For the first three research questions, all medical students who have enrolled in one of three described MOOC integration designs at the LUMC will be invited to participate. For research questions 4 and 5 students will be purposively sampled, based on motivation profiles, SRL scores and integration design, as we aim to have a variety of participants on relevant characteristics to yield insights from various angles. There are no specific criteria for exclusion.
Sample size calculation
We will approach all students in the three cohorts (expected n=20 for A, n=300 for B, and n=18 for C) and expect a response rate of at least 80% for quantitative data collection of RQ1, RQ2 and RQ3, resulting in 16, 240 and 14 participants per design, respectively. For qualitative data collection of RQ4 and RQ5, we will strive to include participants from all three integration designs, and aim to have a diversity in motivation profiles and SRL scores. We anticipate data saturation for both qualitative RQ’s to be reached with a minimum of 8 interviews and a maximum of 16 interviews. We deem the chances of students participating quite high as previously students in these courses have been open to fill in evaluation forms. In addition, other medical education studies that have been performed in the LUMC with medical students have gained highly satisfying response rates.
Methods
Study parameters/endpoints and materials
For each research question the primary study parameters, materials and analyses are described. A summary can be found in table 1.
Research questions, related study measures, time points, data types, potential and expected sample per design, and analyses.
RQ1: What are motivation profiles of medical students in three different MOOC integration designs, and do the three integrated MOOC designs differ in students' motivation profiles?
Parameters: Motivation profiles and association between motivation profiles and integration designs.
Materials: Participants will complete the Learning Self-Regulation Questionnaire,30 adapted to MOOC learning, with statements on controlled and autonomous motivation. The reported Cronbach’s α is 0.80 for autonomous and 0.75 for controlled motivation. To discern motivational profiles we will use a two-step cluster analysis. A double-split cross-validation procedure will be used to examine the stability of the cluster solutions, as described by Vansteenkiste et al.16
Analysis: Assuming we will find at least two different motivational profiles, they will be handled as nominal categorical data. As we have three different groups for the independent variable, a χ2 test will be performed to investigate if specific integration designs are associated with certain motivational profiles.
RQ2: How are psychological needs of medical students satisfied or frustrated in different MOOC integration designs?
Parameter: Variety in need satisfaction and frustration between integration designs.
Materials: The Basic Psychological Need Satisfaction and Frustration Scale31 will be adapted to MOOC learning and yield scores for satisfaction and frustration of the psychological needs of autonomy, competence and relatedness for each participant. Cronbach’s α’s for subscales are between 0.71 and 0.88 for the English version of the questionnaire.
Analysis: The 5-point Likert scales yield numerical data, which we assume will be normally distributed; however this will be checked. Here, need satisfaction and frustration are the dependent variables and a one-way ANOVA (analysis of variance) and post-hoc tests will be performed to investigate the difference between integration designs for average scores of need satisfaction and frustration.
RQ3: What is the relationship between autonomous motivation to learn in an integrated MOOC and self-regulated learning in that MOOC?
Parameters: Autonomous motivation and self-regulated online learning.
Materials: The Learning Self-Regulation Questionnaire30 that will also be used to discern motivational profiles, measures autonomous motivation. Participant data from this questionnaire can thus be reused. The Cronbach’s α is reported to be 0.75 for autonomous motivation. The Self-regulated Online Learning Questionnaire Revised32 will be used to collect participant scores for perceived metacognitive activities before, during and after learning, and for time management, environmental structuring, persistence and help seeking. The Cronbach’s α are reported to be between 0.68 and 0.90 for all subscales.
Analysis: Autonomous motivation and all subscales for SRL are measured with 7-point Likert scales, yielding numerical data which we assume will be normally distributed; however this will be checked. A cross-lagged panel correlation33 will be performed to find the direction of the relationship. As described by Tyagi and Singh33 this analysis necessitates two constructs, X and Y measured at two different points in time, for example, times 1 and 2. The two variables and two points in time (lags) generate four variables (X1, X2, Y1 and Y2) and the four variables generate six correlations: two autocorrelations (rX1X2 and rY1Y2); two synchronous correlations (rX1Y1 and rX2Y2) and two cross-lagged correlations (rX1Y2 and rX2Y1). These correlations will be calculated with a Pearson’s r correlation test. The cross-lagged differential is calculated: rX1Y2 minus rX2Y1. In general, if the cross-lagged differential is positive, the causal predominance is that of X causing Y, and if the cross-lagged differential is negative, the causal predominance is that of Y causing X. Interpretation of results about causality will be guided by the more specific ‘rules’ as posed by Soelberg34 (1967) and Farris35 (1969) for interpretation of cross-lagged panel design results.
RQ4: What processes are involved in goal acceptance or rejection of medical students in integrated MOOC designs with assigned learning goals?
Parameter: Process themes regarding goal acceptance or rejection.
Materials: Semi-structured interviews using an interview guide (online supplemental appendices A and B), and a grounded theory approach analysis will result in qualitative themes with respect to goal acceptance and rejection.
Supplemental material
Analysis: Interview data from the first part of the interview will be analysed in iterative cycles as described in the AMEE guide about grounded theory.36 A coding scheme will be developed with a second investigator, starting with open coding, followed by axial coding and finally selective coding. When the coding scheme is finalised, a third researcher will perform a member check, and the scheme will be applied to all interview data. This process will be facilitated by using qualitative data analysis software (Atlas.ti).
RQ5: What difficulties do students perceive in working with the assigned goals, and what helps them when working with assigned goals?
Parameter: Perceived obstacles when working with assigned online learning goals.
Materials: Semi-structured interviews using an interview guide (online supplemental appendices A and B), and a template analysis approach using Cultural Historical Activity Theory as a template will result in qualitative themes.
Analysis: The second part of the interview data will be analysed with a template based on the components described in Cultural Historical Activity Theory: (1) the objective of the activity system, (2) the actor engaged in the activities, (3) the community or social context, (4) the tools used by actors in the system, (5) the division of labour within the system and (6) rules that shape the system.37 Problems with regard to an activity system, in our case the student learning online with assigned learning goals, can exist within and between these components, or when components from two activity systems meet. With a second investigator open codes for obstacles or problems will be created and discussed, as will their fit with the template. Emerging codes that do not fit with the template will be open, axially and, selectively coded with the second investigator to form new themes, and existing template themes that are not present in the data will be abandoned. When the coding scheme is finalised, a third researcher will perform a member check, and the scheme will be applied to all interview data. This process will be facilitated by using qualitative data analysis software (Atlas.ti).
Other study parameters
Identification number. To be able to link students’ within-subject data, an eight digit identification number will be collected. This will be generated by the students using the first two letters of their first name, the first two letters of their last name, their birth date and month.
Sex, age and university. Will all be handled as possible confounders or covariates for RQ3, and will inform purposive sampling for interviews for RQ4 and RQ5.
Study procedures
Data will be collected between 1st July of 2019 when the first LOTS cohort starts and 31st August of 2020 when interviews have been conducted with participants of the second LOTS cohort, as shown in figure 2. When a student enrols in one of the integration designs, we will be notified and receive the student’s email address. All students that enrol receive an email with an information letter (online supplemental appendix C), an informed consent form (online supplemental appendix C) and compiled questionnaire 1 (T1), which includes the measures for autonomous motivation and SRL, as can be seen in figure 3. After completing the MOOC component of a MOOC integration design, students will again receive the information letter and consent form, and will be asked to fill in compiled questionnaire 2 (T2), which includes measures for autonomous and controlled motivation, SRL and need satisfaction and frustration. According to the MOOC integration design the moments of data collection differ per integration design as can be seen in figure 2.
Study procedures. F2F, face-to-face activities; MOOC, Massive Open Online Course.
Based on motivation profiles, SRL scores, integration design and sex, students will be purposively selected and asked to also participate in interviews. In semi-structured interviews (T3), participants will be asked about two topics: (1) the way they work or do not work with assigned learning goals, and (2) problems or obstacles they face in doing so (interview protocol in online supplemental appendix D). In our view, individual interviews are preferred over group interviews as the processes involved in working with goals and accepting or rejecting them could differ distinctly between students, as may their way of viewing or describing these processes. Students might help each other of thinking about more involved processes in group interviews, but they might also confuse each other. In addition, interviews offer most opportunity for clarifying questions to understand the involved processes. Interviews will take approximately 30 min to 1 hour to complete and will be arranged as face-to-face on a location preferred by the participant, or Skype meetings, depending on the country of residence of the participant. Interviews will be recorded and verbally transcribed.
Withdrawal of individual subjects
Subjects can leave the study at any time for any reason if they wish to do so without any consequences.
Ethical considerations
Regulation statement
This study has been approved by the Educational Research Review Board (ERRB) of the LUMC. This study does not fall under the Dutch Medical Research Involving Human Subjects Act (WMO). However, it is subject to the Dutch General Data Protection Regulation (AVG) and will be conducted according to it.
Recruitment and consent
The first author or another research team member, who has no educational role in relation to the students in these cohorts, will approach students by email to inform them about the opportunity to participate in the study when they have enrolled for the concerned course, but have not started the MOOC part. In addition, a notification will be placed on the Learning Management System (Blackboard). Email addresses will be gathered through the coordinators of each integrated MOOC design course. Students will receive an attachment with extensive information about the research and aspects of their participation (online supplemental appendix C) and an informed consent form (online supplemental appendix C). The information letter will include information on the possibility that participants will be approached to also partake in an interview. At the finalisation the MOOC component of each course, students will be contacted face-to-face before or after they have a workgroup or lecture, or after an exam. They receive the information and informed consent again, which in case of participation will be followed by the questionnaire. Permission from the course coordinators will be obtained for the study to take place during the start or end of the workgroup or lecture, or at the end of an exam. When participants for the interview study have been sampled based on above-mentioned criteria (section 4.2), they will be contacted via email with information about the interview (online supplemental appendix D) and asked to partake. When the interview has been concluded, participants will sign the interview consent form (online supplemental appendix E) to use their interview data, as beforehand it will be difficult to have insight into what will be discussed. For the use of quotes, explicit consent will be asked afterwards.
Benefits and risks assessment, group relatedness
No disadvantages or risks are associated with participating in the study, nor are there direct advantages for students, as will be explicitly stated in the information letter. The only burden would be the 15 to 20 min students will have to spend on the compiled questionnaire. It will be possible for students who are interested to obtain information on their motivation profile. Students who also participate in the interviews will additionally spend 30 min to 1 hour. Participation may lead to significant findings and implications for future integrated MOOC education. Students can leave the study at any time for any reason if they wish to do so without any consequences for their study progress. The collected data will not be traceable to students’ identities after it has been processed.
Incentives
For students that participate in the face-to-face interviews a hot or cold drink and some snacks will be provided.
Administrative aspects, monitoring and publication
Handling and storage of data and documents
The collected data will be processed and coded by the first author using a subject identification code list. Therefore, the research data will not be traceable to an individual student. To ensure data safety, the key file will be stored separately from the anonymised data set on the password-protected personal network storage drive. Only the first author will have access to this document. The anonymised research data will be saved in a SPSS file and stored in a SharePoint Office 365. SharePoint Office 365 is a safe shared Virtual Research Environment within the LUMC according to and recommended by the department of Biomedical Data Sciences. The data set will be accessible only for the research team, mentioned earlier in this proposal. The data will be stored for 10 years for further research purposes according to the ‘Dutch Code of Conduct in Scientific Pursuit’ of the Association of Universities in the Netherlands. In case of withdrawal all collected data of a particular subject will be deleted and removed from the analysis.
Monitoring and quality assurance
The quality of the study is provided by the following criteria:38
Quantitative quality
Internal validity. (1) Pilot of questionnaires: All questionnaires have been previously validated. The small changes to accommodate the MOOC context will be piloted in think-aloud sessions with at least three medical students of similar age to ensure students understand the questions. (2) Use of the same MOOC: by comparing integration designs using the same MOOC, we maximise the validity of the findings regarding differences between integration designs. (3) Check of digital data entry: After data entry into a digital file has been completed, every entry will be checked to ensure the digital raw data file contains no errors.
External validity. (1) To increase replicability, efforts have been made to extensively describe the context of the study, as are the methods. The MOOC teaching modes profile will be disclosed and integration designs are described. As this particular MOOC is open to other institutions for integration, replication should be possible to a high extent. (2) By categorising a MOOC integration design based on a set of relevant characteristics, findings are expected to be more generalisable to other contexts with the same characteristics. The use of a specific MOOC decreases the generalisability to other contexts where other MOOCs will be integrated. As the MOOC teaching mode profile will be disclosed, inferences can be made about similar MOOCs.
Reliability. (1) Internal consistency of instruments will be checked. All questionnaires have been previously validated and Cronbach’s α’s have been reported above and are all 0.67 or higher which we deem acceptable. Cronbach’s α’s will be checked for our sample of participants for each scale when data has been collected.
Objectivity. (1) Participants identities are anonymised, while maintaining the opportunity to link participants results from T1 and T2. (2) The original data will be stored safely to ensure accountability to participants, the research community and the public.
Qualitative quality
Credibility. (1) Respondent feedback will be member checked by communicating preliminary findings to the participants. Consequently, their feedback might generate alternative or new insights. The results will be adjusted accordingly. (2) Researcher triangulation: The interview transcripts will be analysed by at least two researchers independently. (Dis)agreement on emergent findings will be discussed and reported.
Transferability. (1) The learning context and research context will be described in depth to offer meaning to other similar contexts. (2) We will perform purposive sampling in order to obtain a rich diversity in the participant sample and the variety in the interview responses.
Dependability. (1) Data saturation: Saturation is reached if new interviews do not yield any new themes. If saturation is suspected, two more participants will be recruited to verify saturation. (2) Iterative data-collection and analysis: Since this qualitative research comprises of an iterative process, data will continuously be analysed and re-examined. Emerging topics which need further elaboration or clarification will be addressed in subsequent interviews.
Confirmability. (1) Reflexivity: It is likely that many ideas will come up during this study. A laboratory journal will be kept and serve as a tool for keeping track of reflections (personal perspectives, thoughts and assumptions) during the data collection and analyses. (2) Findings will be discussed with peers and experts at conferences. (3) Literature will be searched for findings that contest and or confirm our findings.
General quality
Mixing the methods: All data together offer in depth insight into effectively using SRL skills in integrated MOOC learning (RQ4 and RQ5), how this is influenced by motivation (RQ3) and how this is influenced by the MOOC integration design (RQ1 and RQ2).
The methods complement each other also in the following more tangible ways: (1) the quantitative data support the qualitative data collection, as we sample purposively on extremely relevant variables, and (2) the interviews give depth to the motivation profiles that have been created quantitively.
Data storage and handling. According to the Association of Universities in the Netherlands conduct, data will be stored and saved for 10 years. Data will be destroyed afterwards. Participants have the right to see their data and to request any changes or deletion of the data. In case of withdrawal all collected data of particular subjects will be deleted and removed from the analysis.
Amendments
Amendments are changes made to the research after a favourable opinion by the accredited ERRB of the LUMC has been given. All amendments will be notified to the ERRB.
Temporary halt and (prematurely) end of study report
The first author will notify the accredited ERRB of the end of the study within a period of 8 weeks. The end of the study is defined as the last moment of data collection, which is most likely to be after saturation is reached and participant feedback has been collected for the qualitative part of the study. Analysis and dissemination of findings will continue afterwards, however students participation will have ended. The first author will notify the ERRB immediately of a temporary halt of the study, including the reason of such an action.
Public disclosure and publication policy
Planned dissemination of findings include:
Abstracts for AMEE conference 2020.
Abstracts for EARLI SIG 8 conference 2020.
Abstracts for NVMO conference 2020.
Research article for RQ1 and RQ2, preferably open access.
Research article for RQ3, preferably open access.
Research article for RQ4 and RQ5, preferably open access.
Each article will be the basis of a chapter in the dissertation of the first author.
Acknowledgments
We kindly thank Marjolein Versteeg, Katja Bogomolova, Charlotte den Bakker, and Floris van Blankenstein for helpful discussions regarding the content and structure of this protocol.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors In writing this protocol, RAH (first author) read relevant literature and discussed a list of possible objectives with PdJ (second author) and MR (last author). A first version of the protocol was then written by RAH, and after a few iterations of discussion with PGMjD, a third version was discussed with all authors. Input from this discussion was used to produce a version that was reviewed in great detail by WA (third author) and MR. The final version was then presented to the ERRB by RAH.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.