Evaluating results of the implementation research MOOC using Kirkpatrick’s four-level model: a cross-sectional mixed-methods study

Introduction An implementation research (IR) massive open online course (MOOC) was developed by the Special Programme for Research and Training in Tropical Diseases, to address the scarcity of training in low-income and middle-income countries in the field of IR. The Kirkpatrick model was used to evaluate the IR MOOC as it is widely applied for evaluation of training and educational programmes. The Kirkpatrick model evaluates training programmes on four levels: reaction, learning, behaviour and results. This paper addresses the impact of the IR MOOC on participants’ professional practice. Methods Findings are based on analysis of survey and interview data collected 1.5–2 years after the conclusion of the two 2018 IR MOOC offerings. Of the 3858 MOC participants, 748 responded to the anonymous online survey and seven of these respondents were interviewed. All data are self-reported. Results The IR MOOC was successful in enhancing the professional practice of participants and for their organisations. Over 40% reported modifying or implementing changes in their professional work. Respondents reported that participation in the MOOC had improved their ability to conduct IR, enhanced their professional profiles and increased their opportunities for collaboration, research and job promotion. Respondents stated that the MOOC had improved their work quality and productivity, and allowed them to contribute to research, initiate and develop professional collaborations and train others in IR. Respondents reported an increase in applying for grants and scholarships and presenting and publishing work on IR after participating in the MOOC. Barriers applying the knowledge gained from the IR MOOC were experienced, for example, due to a lack of funding and lack of support from colleagues, managers and organisations. Conclusion Participants perceived that the IR MOOC was successful in its aims of delivering medium-term and long-term results in relation to their own and their organisations’ professional outcomes.

ABSTRACT Introduction An implementation research (IR) massive open online course (MOOC) was developed by the Special Programme for Research and Training in Tropical Diseases, to address the scarcity of training in low-income and middle-income countries in the field of IR. The Kirkpatrick model was used to evaluate the IR MOOC as it is widely applied for evaluation of training and educational programmes. The Kirkpatrick model evaluates training programmes on four levels: reaction, learning, behaviour and results. This paper addresses the impact of the IR MOOC on participants' professional practice. Methods Findings are based on analysis of survey and interview data collected 1.5-2 years after the conclusion of the two 2018 IR MOOC offerings. Of the 3858 MOC participants, 748 responded to the anonymous online survey and seven of these respondents were interviewed. All data are self-reported. Results The IR MOOC was successful in enhancing the professional practice of participants and for their organisations. Over 40% reported modifying or implementing changes in their professional work. Respondents reported that participation in the MOOC had improved their ability to conduct IR, enhanced their professional profiles and increased their opportunities for collaboration, research and job promotion. Respondents stated that the MOOC had improved their work quality and productivity, and allowed them to contribute to research, initiate and develop professional collaborations and train others in IR. Respondents reported an increase in applying for grants and scholarships and presenting and publishing work on IR after participating in the MOOC. Barriers applying the knowledge gained from the IR MOOC were experienced, for example, due to a lack of funding and lack of support from colleagues, managers and organisations. Conclusion Participants perceived that the IR MOOC was successful in its aims of delivering medium-term and long-term results in relation to their own and their organisations' professional outcomes.

BACKGROUND
The past two decades have seen an increasing recognition of the potential implementation research (IR) in improving the effectiveness of health interventions, particularly in low-income and middle-income countries (LMICs). 1 IR intends to identify implementation bottlenecks and inform strategies designed to overcome these barriers for efficacious health interventions that are implemented in real-world settings and scaled beyond highly controlled clinical trials. 2 3 The maturing of IR as a scientific field and its growing popularity has inspired a cadre of implementation scientists across disciplines 4 5 and geographies. 1 6 IR is gaining ground within LMICs in particular, where implementation challenges are often encountered but inappropriately studied. 6 7 The IR massive open online course (MOOC) was designed to address this gap by providing IR training to health professionals and students, particularly those living and working in LMICs. Because each implementation bottleneck is caused by context-specific challenges, local researchers who possess an in-depth knowledge of the community, health  10 Prior studies of the IR MOOC have demonstrated considerable increases in learners' IR knowledge and positive changes to their professional behaviour. 11 It has also been shown that the MOOC is reaching its intended audience of public health and research professionals in LMICs. 11 Lessons learnt since the introduction of the MOOC include the importance of delivering the course in languages other than English, presenting geographically relevant case examples, and training course facilitators on stimulating discussion through the online forum. 12 13 The current study builds on these findings to examine results, that is, if and how participants' learning from the IR MOOC translated into their job outcomes and throughout their organisation, which is considered the ultimate impact of the training and in effect evaluates whether the training achieved its aims.
Evaluating results using Kirkpatrick's model This study evaluates the IR MOOC using the Kirkpatrick Model. 14 The model has been widely used to evaluate courses for learners in high-income settings, [15][16][17][18][19] and is being applied to MOOCs targeting LMICs participants. 20 The Kirkpatrick Model includes four levels of evaluation: (1) reaction, (2) learning, (3) behaviour and (4) results. This paper presents an evaluation of results, where results refers to improvement of job and organisational performance. The findings related to participants' reaction 13 and behaviour 11 have been reported previously.

Survey data
The composition and methodology for the online survey have been described previously. 11 In short, quantitative and qualitative data were collected between November 2019 and February 2020 through an online anonymous survey consisting of 43 open-ended, multiple choice, and 5-point Likert-type questions (ranging from 'to a very small extent' to 'to a very large extent'). The survey captured participants' demographics, satisfaction, change in behaviour, and results from the MOOC that impacted their professional work or that of their organisation. The survey, programmed in Qualtrics software, 21 was sent via email in November 2019 to all participants (N=3858) from the two 2018 MOOC cohorts. Our aim was to include a diversity of perspectives among all who participated in the IR MOOC, not only those who earned the certificate of completion. A response rate of 19.4% (N=748) was achieved, with a 3% margin of error. Descriptive statistics for the quantitative data were calculated using both Qualtrics and Microsoft Excel. 22 Data from the open-ended questions were thematically analysed following Braun and Clarke's 23 six phases: (1) familiarising oneself with the data, (2) generating initial codes, (3) searching for themes, (4) reviewing themes, (5) defining and naming themes, and (6) producing the report. The thematic coding scheme was produced through discussions between author 1 and the primary coder. Author 1 double-coded approximately 10% of the data to check for interrater reliability. Levels of agreement were greater than 90% and inconsistencies were resolved through discussion between the two coders. The reporting of this evaluation is informed by the CRe-DEPTH guidelines. 24 Interview data On completion of the survey, respondents were asked to provide their contact details if they were interested in being interviewed regarding their experiences in participating in the MOOC. Out of 361 who volunteered, a convenience sample of 32 respondents were then contacted via email by author 1 to schedule a teleinterview. Potential interviewees were chosen in order to represent the nationalities, professions, genders and education levels of the MOOC participants. Of those invited, seven responded. They were provided with a plain language statement on the aims of the research project and participants provided written informed consent. Semistructured teleinterviews were conducted in June and July 2020. Nineteen questions built on the survey questions and focused on participants' perceptions of the MOOC, their learning and ways in which the MOOC had impacted on their professional behaviour and their organisations. The semistructured teleinterviews ranged in duration from 22 to 50 min. The teleinterviews were audiorecorded and transcribed. Where relevant, the interview data is used to complement the findings from the survey data.

Public involvement
The public was not involved in the design, conduct, reporting or dissemination plans for this research.

FINDINGS
Demographics of MOOC registrants and survey and interview participants As described previously, 11 3858 registrants participated in the May and October MOOC. Of these, approximately 30% (1163/3858) earned a certificate of completion. Registrants represented 115 different countries, the majority of whom (62.4%) came from the WHO African Region. Participants from the WHO South-East Asian Region accounted for 17.7%, the WHO Americas Region accounted for 9.9%, the WHO Eastern Mediterranean Region accounted for 5.5%, the WHO European Region accounted for 2.8% and participants from the WHO Western Pacific Region accounted for 2.2% Figure 1 displays the countries represented by the MOOC participants. Table 1 provides a summary and comparison of the demographics of the MOOC participants and the survey respondents (also reported in Launois et al 11 ). Overall, the survey respondents are largely representative of the MOOC participants. Table 2 presents the demographics of the seven interviewees. Interviewees were made up of five males and two females. One interviewee held a bachelor's degree and three each held a master's and doctoral degree. Four of the interviewees' age range was 31-35, one was aged between 26 and 30 and two were aged between 46 and 50. Although a roughly equal number of male and female respondents were invited to be interviewed (32 in total: female=43.75%, male=56.25%), note that mainly males responded. Six out of seven participants are living and working in Africa and one is living and working in Germany. Regarding profession, three participants work as university academics. One participant is enrolled in a master's degree in public health. Two participants work for non-governmental organisations, and are involved in public health projects. One participant is a specialist doctor and one is a clinical researcher. In terms of research Open access experience, four participants have been involved in public health research. Two participants have experience in clinical research. One participant's research interest is in capacity building and policy-making.
From the survey, the most commonly cited reasons for participating in the IR MOOC are provided in table 3.
All of the interviewees likewise cited self-motivated reasons for participating in the MOOC. The main reason was that the course was relevant to their job, research interests or educational background. They received information about the course from sources other than their workplace.
Results from participating in the MOOC As a result of the IR MOOC, survey respondents had undertaken a range of professional activities, as shown in table 4. Over one-third had modified or implemented changes in their professional practice and initiated or conducted research, while a quarter had initiated or developed new professional collaborations.
These findings were reflected in the interview data, with four participants reporting that they improved how they conducted their research: I think for me one of the benefits of taking the MOOC was to kind of formalise the way that we approach some of our research more within that framework, to make it more objective and outcome oriented. (Interviewee 7) Three of the interviewed participants stated that they have improved the way they conduct research and solve problems as a result of participating in the MOOC: It [the MOOC] helped me, I think mainly to understand the-to help me to solve one of our other thing-one of our study in Mali and this study was facing a lot of challenge in community and there was fake news, that sort of thing. So MOOC helped me very much to overcome this challenge. (Interviewee 2)

Open access
A majority of surveyed respondents (80.1%) had passed on knowledge gained from the MOOC to colleagues or peers. Four of the interviewees likewise supported this, stating that they had either shared their knowledge with their teams or advised their teams on IR projects: I actually shared some of the information with the rest of my team afterwards. (Interviewee 7) Another participant perceived that passing on knowledge of IR to colleagues was necessary for further collaboration and career prospects in this area: There's a gap in the knowledge because not all of us are really familiar with implementation research so-and those of us who have a good knowledge of it, or let's say a basic knowledge -we try as much as possible to teach other people, train them on implementation research, so that we can work together as a team and achieve our main goal. So for career paths, it's a big challenge because it's just starting and we don't know where it will end up, but it's always good to start so that you know where you are going to. (Interviewee 1)

Opportunities for career enhancement
In relation to whether participation in the MOOC had increased survey respondents' professional opportunities for promotion, just under half (52.4%, F=48.4%, M=55.5%) felt it had while just under half (47.6%, F=51.6%, M=44.5%) disagreed. Of the 218 respondents who elaborated through the open-ended questions, almost one quarter cited improved research performance and just over ten percent cited an enhanced professional profile as contributing factors to possible career enhancement. Many stated that they felt more confident professionally due to an increase in IR knowledge, that they felt able to conduct and contribute to research in the field of IR, and that the MOOC had opened doors for them professionally. The most common themes given in response to these questions are: ► Improved research performance (N=52). ► Enhanced professional profile, professional advantage, recognition, awards and publications (N=25). ► Opportunity for collaboration and research (N=23). ► Job promotion (N=21). ► Success or confidence in applying for or retaining a job (N=20). ► Research grants (N=16).

Improved research performance
Responses to the open-ended survey questions indicated that the MOOC had improved participants' research performance, which played a key role in their career enhancement opportunities. This includes writing research proposals, participating in conferences, addressing challenges in research projects, and contributing to research teamwork in general. One respondent stated:

Opportunity for collaboration and research
Participants felt that the MOOC had provided them with increased opportunities for research and collaboration with others: I am currently discussing with my institute directors to make IR as an additional research unit to our existing structure. (Survey respondent, Public health researcher, Papua New Guinea, Obtained certificate)

Job promotion
Many respondents stated that the MOOC had directly assisted them in achieving job promotions, which included both promotion to a higher position or an increase in salary. One respondent stated: I am now part of a national chapter that is training and creating awareness of implementation research. (Survey respondent, Teacher, Rwanda, Obtained certificate) Success or confidence in applying for or retaining a job A number of respondents felt that the MOOC had provided them with either success or confidence in applying for or retaining a job. For example, one respondent stated: My contract was renewed for an extra year thanks to the skills I implemented in my day-to-day management of the project, which was very satisfactory. (Survey respondent, Public health officer, Cameroon, Obtained certificate)

Research grants
Respondents were successful in attracting research grants as a result of participation in the MOOC, as illustrated by the following statement: Open access I got a grant which I am going to use to design an implementation study. (Survey respondent, Researcher, Uganda, Obtained certificate) Benefits to place of employment The majority of survey respondents (66.3%, F=62.1%, M=69.6%) felt that their participation in the MOOC had benefited their place of employment with 276 respondents elaborating on how this was the case. Benefits included improvements in participants' work, increases in participants' IR knowledge and evaluation skills, and that participants shared the knowledge they gained from the MOOC with others within their organisation. These results were ultimately beneficial and resulted in better outcomes for participants' organisations. The following outlines the themes of these responses. ► Improved work quality, efficiency or productivity (N=94). ► Contribution to research activities and collaboration (N=35). ► Teaching, training, supervising, mentoring or advising students or colleagues (N=33). ► Application of the knowledge and skills gained (N=29). ► Enhanced knowledge (N=24).
Improved work quality, efficiency or productivity Respondents cited improvements in work quality, efficiency and productivity. These improvements were found at both the individual and institutional level. The following statement reveals one respondent's perception: Better research questions have been developed and implemented which has led to improvement of quality of services we provide. (Survey respondent, Public health officer, Uganda, Obtained certificate)

Contribution to research activities and collaboration
Participation in the MOOC contributed to respondents' research activities and collaboration, including establishing or expanding collaboration and partnerships at their place of work. Representative statements include: It has enabled extension of new professional collaboration research partners and nations especially in the field of IR to our institute. (Survey respondent, Public health researcher, Papua New Guinea, Obtained certificate) Yes, the training has increased my opportunities. I am part of a research team that we have set up a kind of NGO for Health-Environment and Climate Change. As a result, I bring a lot of research opportunities in tenders. I also contribute to the development of projects looking for funding. (Survey respondent, Teacher, Côte d'Ivoire, Obtained certificate) Teaching, training, supervising, mentoring or advising students or colleagues A benefit of the MOOC cited by respondents concerned their improved abilities in teaching, training, supervising, mentoring or advising students or colleagues, as illustrated by the following quotes: I am able to advise on ways to engage all stakeholders including beneficiaries. (Survey respondent, Student, Malawi, Obtained certificate) I am mentoring junior colleagues in my place of work and together we will increase the productivity and efficiency. (Survey respondent, Public health researcher, Nigeria, Obtained certificate) Application of knowledge and skills gained Respondents stated they were able to apply the knowledge and skills gained from the MOOC in their professional work. One survey respondent applied the knowledge daily: I apply the knowledge gained from the course on a daily basis. (Survey respondent, Administrator, Ghana, Obtained certificate) An interviewee outlined how the MOOC had assisted them in their work: So it [the IR MOOC] has really, really helped me. Right now I can identify how to assess interventions, how to define the implementation problem, how to design implementation challenges, and also test implementation strategies. (Interviewee 1)

Barriers applying IR knowledge
Over 40% of survey respondents (41.5%, F=37.6%, M=44.4%) stated they had faced challenges or barriers to applying the knowledge gained from the MOOC in their professional work or research. Of those surveyed, 217 participants elaborated on their response. The most commonly cited themes are as follows: ► Lack of funding, finance, time and other resources (N=59). ► Lack of knowledge or interest from colleagues, teammates and peers (N=38). ► Lack of support from colleagues, managers, supervisors, organisations, institutions, or workplaces (N=24). ► Limited or no opportunities to apply IR (N=24).

Lack of funding, finance, time and other resources
The most commonly cited barrier was a lack of funding, finance, time and other resources, as illustrated in the following: Funding and getting buy in from policy makers is usually not easy. (Survey respondent, Public health researcher, Nigeria, Obtained certificate) Lack of knowledge or interest from colleagues, teammates and peers Respondents cited lack of knowledge or interest from colleagues, teammates and peers as a common barrier Open access to applying the knowledge and skills from the MOOC in their work: Few people really know about this research, so if you explain others do not understand. (Survey respondent, Public health researcher, Tanzania, Obtained certificate) An interviewed participant stated that a barrier to applying IR, in addition to limited opportunities for collaboration and mentorship, was due to a lack of training opportunities: Also training opportunities-where there would be a kind of hands-on experience. Hands-on experience. Because like I said earlier, in our countries, we have problems of training and retraining. Our research requires training and retraining. It requires application directly in the field. Not on papers or in the classrooms. So that is actually what we are facing.

(Interviewee 6)
Lack of support from colleagues, managers, supervisors, organisations, institutions or workplaces Respondents felt there was a lack of support to use the knowledge and skills of IR gained. This lack of support was often discussed in terms of resistance and in connection with a lack of knowledge from colleagues, managers, supervisors, organisations, institutions, or workplaces. One surveyed respondent stated: Knowledge of IR in Uganda is still low, so application of IR concepts faces resistance from other stakeholders. (Survey respondent, Public health researcher, Uganda, Obtained certificate) These barriers were also found in relation to collaboration and teamwork: Sometimes it is difficult to collaborate with other people in research in developing IR proposals. (Survey respondent, Public health researcher, Cameroon, Obtained certificate) Limited or no opportunities to apply IR Respondents stated they had limited or no opportunities to apply IR in their work due to, for example, the limitations of their current professional positions, as illustrated in the following statement: I did not yet have an opportunity/a project to apply the knowledge gained from the MOOC. (Survey respondent, Public health researcher, Côte d'Ivoire, Obtained certificate)

DISCUSSION
The findings reveal that respondents perceived clear benefits from participating in the IR MOOC in relation to the professional activities they subsequently undertook and the results these had for their organisations. Activities included implementing changes to practice, initiating or conducting research, initiating and developing new collaborations, and developing publications and grant applications. Organisational results included benefits to work quality, efficiency or productivity, increased research activities and collaboration, and transfer of knowledge through teaching, training and mentoring.
Findings also reveal, however, that respondents perceived a lack of support for IR practice by supervisors, colleagues and organisations, in terms of funding, support and mentoring. Other commonly cited barriers by respondents include (1) a lack of understanding of IR by colleagues and the wider community, (2) resistance to change by colleagues, supervisors, policy-makers and organisations, (3) the ability to identify suitable collaborators, (4) bureaucracy and administrative and structural barriers. A number of potential solutions to these challenges could be piloted in the future. Running the IR MOOC in collaboration with donors, such as the Global Fund, could increase the availability of financial support for participants as donors increase their emphasis on IR projects. Mentorship programme for IR trainees are under development now, with the aim of increasing technical support to newly trained investigators. These programmes, if successful, should be institutionalised within training initiatives to help ensure the skills developed through the IR MOOC are applied. Organisational readiness for change is crucial for successful implementation of innovations and evidence-based practice in healthcare and human services. 25 Studies reveal a relationship between a health organisation's culture and social context and clinicians' attitudes towards and adoption of evidence-based practice. 26 Through this study and others, 11 13 the IR MOOC has been demonstrated to be a valuable tool in communicating IR concepts to learners, engaging a diverse set of participants, and catalysing professional change at the individual and organisational level. Importantly, research is now underway on how to best address barriers around institutional demand for IR and need for professional networks. The next generation of training models can use the IR MOOC as a component of a suite of training tools that strengthen the skills of both individuals and teams in combination with practiced-based learning and guided mentorship. 27

Study limitations
A key limitation is that this evaluation is based on the two first offerings of the MOOC and does not include later iterations, including those offered in French or Spanish. Other limitations to this study include response bias, as there may be differences in those who responded to the survey and interview requests compared with those who do not. 28 Further, as survey and interview respondents are highly self-selective, this may be linked to learners with autonomous motivations for behaviour. 29 30 Additionally, validity issues may be present, as data are self-reported Open access by participants. A final limitation is the lack of a control group. The authors acknowledge that the results would be strengthened by interviewing employers and organisations to demonstrate the results and impact reported on in this study.
Contributors PL and DM conceived of the paper. BR led data collection, data analysis, and drafted the initial form of the paper with assistance from MJP. PL, DM, EC, BR and MJP all made significant contributions to the conceptual framework and read and modified drafts. All authors (PL, DM, EC, BR and MJP) read and approved the final manuscript. BR is the guarantor for this article.
Funding TDR funded the development of the MOOC. Grant number: not applicable.
Map disclaimer The inclusion of any map (including the depiction of any boundaries therein), or of any geographic or locational reference, does not imply the expression of any opinion whatsoever on the part of BMJ concerning the legal status of any country, territory, jurisdiction or area or of its authorities. Any such expression remains solely that of the relevant source and is not endorsed by BMJ. Maps are provided without any warranty of any kind, either express or implied.
Competing interests BR and MJP declare that they have no competing interests. PL, DM and EC work for TDR, which funded the development of the MOOC.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Patient consent for publication Not applicable.
Ethics approval Ethics approval for this project was granted by MUHREC at Monash University (project 24065). Participants gave informed consent to participate in the study before taking part.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data are available on reasonable request. The datasets used during the current study are available on reasonable request.
Open access This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/ licenses/by/4.0/.