Table 2

Evaluation framework methodology based on the RE-AIM and Kirkpatrick models—measures, data collection methods, timeline points

Evaluation model dimensionsIndicatorsCorresponding research questionOutcome measuresData collection methodsTimepoint of assessment
EffectivenessKnowledge and perceived confidence in QIPrimary research question—
To what extent does the MOOC improve learner’s knowledge and understanding of QI approaches and increasing perceived confidence in participating in QI initiatives?
Knowledge assessment—subjective/self-report and objective.Survey itemsPre-MOOC and post-MOOC (immediate)
ReachLearners’ characteristicsSecondary research question—
What are the characteristics of the learners taking the MOOC?
Sociodemographic information—age, country, gender, language, education level, employment.Survey itemsPre-MOOC
ReactionSelf-efficacy and motivationSecondary research question—
How did learners react to the course?
Reasons for doing the MOOC.
Self-efficacy in learner’s ability to dedicate time and complete the course.
Survey items
Qualitative interview data
Pre-MOOC and post-MOOC (immediate)
Satisfaction and relevanceSatisfaction with learning experience and relevance to practice.
How participants valued the course—strengths and weaknesses, areas of improvement.
Feedback on course content, layout, format and design.
LearningCollaborative learningSecondary research question—How did the learners learn and how did they engage with other learners?Attitudes and experiences of engaging with others on the course and asking for help. Collaborative learning—advantages and disadvantages.Survey itemsPre-MOOC and post-MOOC (immediate and 3-month follow-up)
Higher order learningPerceptions of whether higher order learning was achieved during the course—apply new information to new situations, acquired new knowledge and understanding of QI.Qualitative interview data
Reflective and integrative learningConnected their learning to problems that could be addressed by QI, better understanding of how a QI problem might look from another person perspective (eg, patient), learnt something that changed the way they understood a concept or idea, connected ideas from the course to prior knowledge and experience.
CapabilityThe degree to which participants acquire the knowledge and confidence to engage in QI efforts based on their participation in the MOOC.
Maintenance (sustainable)Long-term effects of the MOOCSecondary research question—
What evidence suggests that learners retained knowledge acquired from the course?
Learner’s confidence in their ability to design, implement, sustain QI activities.Survey items
Qualitative interview data
(immediate and 3-month follow-up)
BehaviourPostcourse practices in work environment and professional practiceSecondary research question—What evidence suggests that the MOOC increased participation in QI initiatives?Perceived self-efficacy, motivation, confidence in initiating/implementing QI activities.
Impact of the MOOC on work/practice and ability to influence others in QI.
Barriers and facilitators to implementing QI projects.
Survey items
Qualitative interview data
Post-MOOC (immediate and 3-month follow-up)
  • MOOC, massive open online course; QI, quality improvement.