Article Text

Download PDFPDF

Outcomes of the Northern Ontario School of Medicine's distributed medical education programmes: protocol for a longitudinal comparative multicohort study
  1. John C Hogenbirk1,
  2. Margaret G French1,
  3. Patrick E Timony1,
  4. Roger P Strasser2,
  5. Dan Hunt2,3,
  6. Raymond W Pong1
  1. 1Centre for Rural and Northern Health Research, Laurentian University, Sudbury, Ontario, Canada
  2. 2Northern Ontario School of Medicine, Laurentian University, Sudbury, and Lakehead University, Thunder Bay, Ontario, Canada
  3. 3Association of American Medical Colleges, Washington DC, USA
  1. Correspondence to John C Hogenbirk; jhogenbirk{at}


Introduction The Northern Ontario School of Medicine (NOSM) has a social accountability mandate to serve the healthcare needs of the people of Northern Ontario, Canada. A multiyear, multimethod tracking study of medical students and postgraduate residents is being conducted by the Centre for Rural and Northern Health Research (CRaNHR) in conjunction with NOSM starting in 2005 when NOSM first enrolled students. The objective is to understand how NOSM's selection criteria and medical education programmes set in rural and northern communities affect early career decision-making by physicians with respect to their choice of medical discipline, practice location, medical services and procedures, inclusion of medically underserved patient populations and practice structure.

Methods and analysis This prospective comparative longitudinal study follows multiple cohorts from entry into medical education programmes at the undergraduate (UG) level (56–64 students per year at NOSM) or postgraduate (PG) level (40–60 residents per year at NOSM, including UGs from other medical schools and 30–40 NOSM UGs who go to other schools for their residency training) and continues at least 5 years into independent practice. The study compares learners who experience NOSM UG and NOSM PG education with those who experience NOSM UG education alone or NOSM PG education alone. Within these groups, the study also compares learners in family medicine with those in other specialties. Data will be analysed using descriptive statistics, χ2 tests, logistic regression, and hierarchical log-linear models.

Ethics and dissemination Ethical approval was granted by the Research Ethics Boards of Laurentian University (REB #2010-08-03 and #2012-01-09) and Lakehead University (REB #031 11-12 Romeo File #1462056). Results will be published in peer-reviewed scientific journals, presented at one or more scientific conferences, and shared with policymakers and decision-makers and the public through 4-page research summaries and social media such as Twitter (@CRaNHR, @NOSM) or Facebook.

  • medically underserved areas
  • medical students
  • internship and residency
  • social responsibility

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See:

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Strengths and limitations of this study

  • The Northern Ontario School of Medicine (NOSM) is one of a few medical schools in the world with an explicit social accountability mandate employing a distributed medical education model.

  • The study started with the opening of the medical school in 2005 and includes all cohorts as they participate in NOSM's undergraduate (UG) or postgraduate (PG) medical education programmes.

  • Longitudinal tracking allows learners’ educational experience to be matched with intended and actual behaviours (eg, intended vs actual medical discipline) for at least 5 years into independent practice.

  • Natural comparison groups are used to investigate the effect of NOSM admission criteria and educational experience.

  • Data from overlapping surveys and administrative data will cover most gaps arising from missed surveys or low survey response rates.


The Northern Ontario School of Medicine (NOSM), which first enrolled medical students in 2005, is a key initiative in the physician human resources plan of the Province of Ontario, Canada,1 and is an important strategy2 to overcome the long running shortage of medical doctors (MDs) in Northern Ontario.3–5 NOSM's mission statement includes a mandate to be socially accountable to the needs and the diversity of the populations of Northern Ontario’, and to actively involve the ‘Aboriginal, Francophone, remote, rural and underserviced communities’ of Northern Ontario.6 NOSM seeks to increase ‘the number of physicians and health professionals with the leadership, knowledge and skills to practice in Northern Ontario’.

NOSM's approach is based on evidence that if medical schools select learners who have lived in underserved areas such as rural and Northern Ontario and train them in a positive manner in similar environments, then these learners are more likely to practice in these areas. This evidence comes from Canada,7 Ontario,8 Northern Ontario9 ,10 and is synthesised at the international level in several systematic reviews.11–14

Northern Ontario has over 800 000 km2, an area larger than France, and a population density that averages 1 person/km2 with approximately 56% of the population clustered in and around five of the larger urban areas (Timmins, North Bay, Sault Ste Marie, Thunder Bay and Greater Sudbury), which range in size from 43 000 to 161 000 people.15 Northern Ontario includes a larger proportion of two cultural-linguistic minority groups than the province as a whole. Francophones represent 18% of Northern Ontarians versus 5% in the province and Aboriginal people represent 14% vs 2%, respectively.16 ,17 Northern Ontarians have poorer access to and lower use of medical care services than the rest of Ontario.5 ,18 ,19 People in Northern Ontario also have poorer health status than the rest of Ontario, and the health status of Francophone and Aboriginal people is worse.20–23

The Centre for Rural and Northern Health Research (CRaNHR) in conjunction with NOSM and funded by the Ontario Ministry of Health and Long-Term Care (MOHLTC) has tracked learners since 2005, the year in which NOSM admitted its first cohort of undergraduate (UG) medical students.24 The study's objective is to understand how NOSM's socially accountable admission criteria and medical education programmes set in rural and Northern Ontario communities affect choice of medical discipline, practice location, medical services and procedures, inclusion of medically underserved patient populations and practice structure (eg, solo, interprofessional team).

This tracking study is unique as NOSM is one of a few medical schools in the world with an explicit social accountability mandate25 and with medical education provided in communities away from large cities and regional hospitals.24 There is emerging global interest in how well NOSM and similar schools can fulfil their mandates.26 ,27 For example, the Training for Health Equity Network is a worldwide movement of schools committed to improving health equity by transforming education of health professionals.26 Eleven schools in nine countries are committed to measuring how well they match educational outcomes to the needs of the areas they serve.28 A second unique aspect is that the study started with the opening of the medical school and includes all cohorts as they participate in NOSM's UG or postgraduate (PG) medical education programmes. Third, longitudinal tracking allows learners’ educational experience to be matched with intended and actual behaviours (eg, intended vs actual medical discipline) as learners are tracked from the time of their arrival at NOSM and continuing for at least 5 years into independent practice. This is important, as previous and ongoing research demonstrate the utility of longitudinal tracking studies linking admission criteria, medical education and other factors with outcomes.29 ,30 A strength of the study resides in the use of natural comparison groups to investigate the effect of NOSM admission criteria and educational experience. Six groups are defined on three dimensions: (1) learners’ medical school (ie, NOSM vs other medical schools); (2) medical education level (ie, UG vs PG) and (3) medical discipline (ie, family medicine vs other specialties). This paper describes methods developed since the study started in 2005.

Methods and analysis

Study design, participant recruitment and data collection

All learners are tracked through administrative databases (eg, medical school admissions and educational programmes databases, medical licencing agencies registration databases), which provide basic demographic data (eg, table 1), details of the learner’s educational experience at NOSM and selected information on outcomes (eg, provincial health insurance (billing) databases). Additional demographic data as well as the learner's perspective on factors that influence key outcomes plus detailed information on the outcomes are obtained by surveys or interviews.

Table 1

Demographic characteristics of Northern Ontario School of Medicine UG medical students and PG residents

Table 2

Primary research questions, study outcomes and explanatory variables

This prospective comparative longitudinal study follows multiple cohorts from entry into NOSM's UG or PG programmes, and at least 5 years into independent (fully qualified) practice. A purposive sampling strategy invites all NOSM UG and PG medical learners to voluntarily participate in surveys or interviews. Every year, 56 UG students (64 since 2010) are tracked throughout their UG education and into PG residency, when they are joined by additional 40–60 PG residents who are new to NOSM. NOSM UG students are asked to participate in surveys and interviews at the first-year orientation, end of second year, and end of fourth year (figure 1). NOSM PG residents are invited to participate in surveys during orientation and just prior to completion of residency. NOSM UG students who go elsewhere for their PG residency training (30–40 residents per year go to other medical schools for PG training, included in the count of NOSM UGs) are invited to continue their participation in surveys at entry and completion of their residency. To summarise, the study tracks learners who finish (1) their UG education and PG training at NOSM, (2) only their UG at NOSM and go to other schools to complete their PG training or (3) only their PG at NOSM having completed their UG medical education at other schools. These three groups are followed for residents who become family physicians or other specialists (combined) to yield six groups for comparison.

Figure 1

Flow of medical learners through the NOSM as of January 2014 (CFPC, College of Family Physicians of Canada; NOSM, Northern Ontario School of Medicine; PG, postgraduate; PGE, postgraduate entry survey; PGX, postgraduate exit survey; PGY, postgraduate year; RCPSC, Royal College of Physicians and Surgeons of Canada; UG, undergraduate; UGE, undergraduate entry survey; UGM, undergraduate midway survey; UGX, undergraduate exit survey; UGY, undergraduate year).

CRaNHR researchers invite NOSM medical students, but not NOSM personnel, to a CRaNHR-sponsored meal, to explain the study and distribute the survey in paper or electronic format, or by a web page link. Wherever possible, a similar event is organised for PG residents at NOSM. NOSM UGs who go elsewhere for their residency training are invited by email or mail to start or continue their participation in the study. All subsequent contact with participants is by email or mail. A modified tailored design method31 is used for all surveys with at least two reminders, excepting those who have opted out. Each UG survey round lasts until all learners have responded or for 3 months, whichever comes first. PG survey rounds last up to 4 months and are initiated throughout the year because of staggered starts and exits due to, for example, parental leave or extra training requirements for internationally trained medical graduates. Since, contact information for residents at other medical schools can be difficult to obtain, we send an invitation whenever we have updated contact information. Participants can complete an on-line questionnaire, electronic MS Word document or paper form. For each survey round, a draw is held for a $C50 gift card from a national retail store as an incentive to participate.

During the UG entry survey, students in the first 5 years were also invited to participate in short-duration semistructured interviews in their first year and again in their fourth year. Interviews were conducted face-to-face, by telephone, or by Skype, depending primarily on learner preference. All interviews were digitally recorded with the interviewee's permission and conducted by Dr Hoi Cheu (CRaNHR Faculty Investigator) using a six-question interview guide, with prompting questions as needed. Questions were shared at least 1 day prior to the interview. Interviewees were given an honorarium of a $C25 gift card from a national retail store.


NOSM's UG and PG admissions criteria and medical education programmes comprise the exposure. NOSM serves as the Faculty of Medicine of Laurentian University in Sudbury (2011 census metropolitan area population: 161 000) and of Lakehead University in Thunder Bay (2011 census metropolitan area population: 122 000)—located 1000 km apart by road. NOSM selects medical school (UG) applicants with a grade point average (GPA) of ≥3.0 of 4.0 in science and non-science university degrees, and does not require the Medical College Admission Test.32 Mean GPA was 3.8 for NOSM students starting in 2015, and this falls within the 3.7–3.9 range for all other Canadian medical schools.33 Preference is given to students from northern, rural, remote, Aboriginal or Francophone backgrounds so as to reflect Northern Ontario demographics. Learners must also have a strong interest in the understanding of, and aptitude for, practising medicine in Northern Ontario. NOSM provides medical learners with educational and clinical experiences in different health service settings in over 90 rural, remote and northern communities.34 ,35 For instance, all first and second year medical students undertake a 1-month Integrated Community Experience in Northern Ontario Aboriginal and rural or remote communities. In the third year, all medical students complete an 8-month Longitudinal Integrated Clerkship, grounded in family practice, and located in 1 of 15 large rural or small urban communities in Northern Ontario, away from Sudbury or Thunder Bay. Similarly, NOSM's PG residency programmes combine learning at the regional hospitals in Sudbury and Thunder Bay with clinical rotations throughout rural and Northern Ontario. NOSM offers PG residency training in family medicine and in eight additional specialist programmes.36 All of this is designed to select learners from rural areas or who are aware of the healthcare needs of the rural underserved, and enable learners to be trained and mentored by physicians who have chosen to live and practice in Northern Ontario so as to prepare learners for practices with fewer resources than in major population centres.

Research questions, study outcomes and explanatory variables

The main research questions and key variables were derived from the literature with selected input from the funder (MOHLTC). Questions and variables were outlined in a research framework adopted by the advisory committee, updated annually and critically reviewed in the 5th and 9th study year. All tools and methods are being reviewed in 2015—the 11th study year. The main research outcomes (table 2) include:

  • Clinical and organisational practice characteristics: medical discipline, medical services and procedures, patient population, practice organisation (eg, solo, interprofessional care team);

  • Practice location: categorised by geographic region, population size/density and rural–urban continuums.

Explanatory variables include:

  • Learner traits: selected socioeconomic and education demographic characteristics including rural or northern background and language/culture/ethnicity.

  • Medical education: level (ie, UG or PG) and medical school (ie, NOSM or other school).

  • Influential factors: opportunity, personal, familial and societal imperatives that affect decision-making around the main outcomes.

Many study outcomes are collected first as intention and then as actual outcome (eg, intended and actual medical discipline). Intended influential factors are those considered by the respondent as important prior to decision-making, and actual influential factors are those that respondents report in hindsight as having affected their decision.

Development and assessment of study tools

Tools to extract administrative data, questionnaires and interview guides were developed as the charter class progressed through their medical education: UG entry questionnaire and interview guide were developed in the academic year 2005/2006; UG midway questionnaire in 2006/2007; and the UG exit questionnaire and interview guide in 2008/2009. PG residents were tracked since 2009 by using administrative data. However, funding delays meant that the PG entry and exit surveys were not developed until 2011/2012. Measures to fill this data gap are described in the section on limits and strengths.

Operationalisation of outcomes, linkages among independent and dependent variables as well as question wording were based on the literature available when the study began,11 informed by a workshop to evaluate the impact of medical education initiatives in Canada,46 and updated with literature being published as the study progressed.12–14 To provide additional content validation and to facilitate comparisons with other medical educational programmes, most questions were based on similar CRaNHR studies.47–49 Other questions were based on the literature, including a block of questions to measure student attitudes on working and living in rural areas,50 modified to the Canadian context and used with permission (Adams ME, Dollard J, Hollins J and Petkov J, personal communications, 2005). Questions from earlier studies were revised to reflect choices available to NOSM learners. NOSM UG medical students, PG residents and practising MDs (two of each and all located in Sudbury, Ontario) reviewed surveys for content validity and readability. Interview questions inquire about selected key outcomes and related decision-making in greater detail.

Multiple data sources (eg, surveys, interviews or administrative data) for several outcomes improved content validity and allowed checking consistency of response. Test–retest reliability of the questionnaires was not assessed because the research team judged that the likelihood of respondents remembering their answers would be too high over the short term, and that answers to many questions would be expected to change in as little as a few months as respondents became immersed in NOSM's distributed medical education programmes.

Dealing with potential bias in surveys and interviews

To reduce social desirability bias, CRaNHR researchers ensure that NOSM faculty or staff are absent during surveys or interviews. Learners are told that their responses will not affect their academic standing and that only aggregate data would be published or shared with NOSM and other stakeholders. Researchers seek to reduce non-response bias by providing multiple mediums (ie, paper, electronic, or online surveys) for up to 3–6 months, to facilitate participation at the learners’ convenience. Recall bias may be an issue only for selected questions about the geographic location of where respondents or their spouses have lived previously. Researchers use administrative data to assess non-response and recall bias for selected information on demographics and outcomes.

Analytical approach

Data comprise multiple measurements on individual learners generated from an (in)complete census of each cohort, and therefore, descriptive statistics or randomisation tests will be used to determine associations or group differences.51 ,52 The χ2 tests, logistic regression and hierarchical log-linear models will be the most frequently used statistical methods. Cohorts are stratified by medical school (ie, NOSM vs other), education level (ie, UG vs PG) and medical specialty (ie, family medicine vs other specialties). The use of multiple imputation techniques to handle missing data will be considered in the context of the specific analytical method or research question. Every effort will be made to contact non-respondents, provided they have not explicitly declined to participate. Administrative data are used to track learners, obtain basic demographic information, details of the learners’ NOSM education experience as well as selected outcomes, while surveys and interviews allow for collection of more detailed data.

Interview transcripts and responses to open-ended questions are analysed using an iterative analytical and inductive approach to group findings within each question.53 Transcripts and electronic recordings are re-examined to ensure that context is preserved, and that confirmatory and contradictory findings are noted. Researchers’ interpretations are distinguished from key informants’ statements,54 while anonymised quotes illustrate the scope and depth of groupings plus exceptions, if any.


All data are stored on a secure server hosted by Laurentian University with access to individual-level data restricted to CRaNHR researchers directly involved in the study. CRaNHR shares only aggregated data (cell size >5) with NOSM personnel or other stakeholders and researchers, and follows other Statistics Canada guidelines to reduce identity, attribute or residual disclosure.55

Results will be published in peer-reviewed scientific journals and presented at one or more scientific conferences. Research highlights will also be shared with policymakers and decision-makers and the public through 4-page reader-friendly summaries of research results (Research In FOCUS On Research), and by social media such as Facebook ( and Twitter (@CRaNHR, @NOSM and researchers’ accounts).

Limits and strengths of the approach

One limitation arises in assessing the exposure because NOSM selects UG medical students (but not necessarily PG learners) with rural or northern Canada backgrounds. Given that rural background is strongly associated with practice in rural areas,7 ,8 ,11–14 there may not be much variation remaining among NOSM medical students to predict outcomes such as location of rural practice. However, the evidence for the influence of other factors, such as northern Canada background, language/culture, gender or marital/partnership status, on outcomes such as medical discipline and practice location varies among contemporary studies11–14 and may be evolving over time and so the study will assess these influences. In addition, the tracking study is able to isolate the influence of different medical schools (ie, NOSM vs other) at different levels (ie, UG vs PG), and for different medical disciplines (ie, family medicine vs other specialties).

Small population size may limit some analyses given that there are 56 new UG students each year (64 since 2010) and a lower number in some PG programmes, especially specialties other than family medicine. Groups may be combined to achieve adequate numbers for analysis, albeit at the loss of some detail.

Choice of outcome measures derived from medical care needs of Northern Ontario and situated in the political context may be interpreted as a limitation as well as a strength. Perhaps a more important limitation is that study outcomes (ie, practice location and scope of practice) are proxies of the ultimate outcome—the health of Northern Ontarians. However, choice of proxy outcomes is reasonable given that NOSM is an important step in ensuring that there are sufficient numbers of skilled and locally trained MDs in Northern Ontario.1 ,2 The expectation is that improved access to MDs will help improve the health of Northern Ontarians.

Other limitations include delays and gaps in execution of surveys. UG surveys and interviews have been on schedule since early 2006 (a 6-month delay), while PG surveys have been on schedule since 2012 (2 prior cohorts had incomplete coverage). Fortunately, administrative data is available from NOSM, and missed PG entry surveys had near-temporal equivalents in the UG exit survey, and so gaps in PG entry survey coverage exist only for PGs new to NOSM in 2009 and 2010. Changes in the wording of questions, or response options, create challenges for temporal continuity that are addressed by a detailed codebook that facilitates appropriate comparisons and provisos.

Study tools and methods are reviewed in-house, which increases internal utility, but may reduce external validity. Although there is no third-party review, many of the indicators and outcomes are copied or derived from the international literature. In addition, several advisory committee members are experts in rural or distributed medical education in Canada, the USA and Australia, and the study benefits accordingly.

Future study

The tracking study will be integrated within a broader research programme assessing the medical, social and economic impact of NOSM on Northern Ontario communities building on previous research.56 Detailed individual-level data allows for investigations into the relationship between specific aspects of NOSM's programmes and medical education outcomes or socioeconomic impacts. For instance, practice characteristics (eg, medical discipline, geographic location) of medical students with science backgrounds could be compared with students with non-science backgrounds.57 Other examples would be to compare performance and practice characteristics of students who had their third year clerkship in larger versus smaller communities (Ellaway RH and Graves L, personal communication, 2011), or to assess the effect of cultural safety training (Jacklin K and Maar M, personal communication, 2012). The integrated study will include investigations into NOSM's admission criteria and processes.


This paper describes a prospective, comparative, multicohort, longitudinal study of NOSM UG and PG medical learners that tracks learners as they progress through the medical education system beginning at admission into NOSM and at least 5 years into independent practice. The tracking study also serves as a platform upon which other research can improve understanding of the role of learner background and medical education experience on outcomes germane to the health and well-being of people living in sparsely populated and medically underserved areas such as Northern Ontario.


The authors greatly appreciate the ongoing participation of medical learners and practising physicians in our study. They also thank the Northern Ontario School of Medicine (NOSM) and Centre for Rural and Northern Health Research (CRaNHR) colleagues, especially past and present members of the Tracking Study Advisory Committee, for their advice and support. The authors thank their Australian colleagues50 for permission to modify and use their questionnaire. The views expressed in the publication are those of the authors and do not necessarily reflect that of the Ontario Ministry of Health and Long-Term Care or NOSM.



  • Twitter Follow the Centre for Rural and Northern Health Research at @CRaNHR and the Northern Ontario School of Medicine at @NOSM

  • Contributors JCH contributed to the study design and tools, study administration, collection, analysis and interpretation of data, and also writing the paper. MGF contributed to design of the tools, data collection and editorial review. PET contributed to the study design and tools, data collection, interpretation and editorial review. RPS provided project leadership and contributed to the study design and tools, data interpretation and editorial review. DH advised on the study design and tools, data interpretation and editorial review. RWP contributed to the study design and tools, data interpretation and editorial review.

  • Funding This work is supported by the Ontario Ministry of Health and Long-Term Care (Grants 04254SB/2005/A and NOSM Tracking Study—January 18.1 2011).

  • Competing interests JCH works part time for NOSM as a research tutor; DH was formerly, while RPS is currently employed full time by the Northern Ontario School of Medicine. JCH, PET and MGF receive partial salary support from the provincial government grants.

  • Ethics approval Ethical approval was granted by the Research Ethics Boards of Laurentian University (REB #2010-08-03 and #2012-01-09) and Lakehead University (REB #031 11-12 Romeo File #1462056) starting in 2005, and has been renewed annually or as new tools were produced.

  • Provenance and peer review Not commissioned; externally peer-reviewed.

  • Data sharing statement Conditions of our ethical approvals permit the Centre for Rural and Northern Health Research to share only aggregated data with NOSM personnel, stakeholders or other researchers.