Article Text

Download PDFPDF

Original research
How do trainee doctors learn about research? Content analysis of Australian specialist colleges’ intended research curricula
  1. Paulina Stehlik1,2,
  2. Christy Noble3,4,
  3. Caitlin Brandenburg5,
  4. Peter Fawzy6,
  5. Isaac Narouz6,
  6. David Henry1,2,
  7. Paul Glasziou1
  1. 1Institute for Evidence-Based Healthcare, Bond University, Gold Coast, Queensland, Australia
  2. 2Evidence Based Practice Professorial Unit, Gold Coast Hospital and Health Service, Southport, Queensland, Australia
  3. 3Allied Health, Gold Coast Hospital and Health Service, Southport, Queensland, Australia
  4. 4School of Medicine, Griffith University, Gold Coast, Queensland, Australia
  5. 5Office for Research Governance and Development, Gold Coast Hospital and Health Service, Southport, Queensland, Australia
  6. 6Faculty of Health Science and Medicine, Bond University, Gold Coast, Queensland, Australia
  1. Correspondence to Dr Paulina Stehlik; pstehlik{at}bond.edu.au

Abstract

Objectives Patients do better in research-intense environments. The importance of research is reflected in the accreditation requirements of Australian clinical specialist colleges. The nature of college-mandated research training has not been systematically explored. We examined the intended research curricula of Australian trainee doctors described by specialist colleges, their constructive alignment and the nature of scholarly project requirements.

Design We undertook content analysis of publicly available documents to characterise college research training curricula.

Setting We reviewed all publicly accessible information from the websites of Australian specialist colleges and their subspecialty divisions. We retrieved curricula, handbooks and assessment-related documents.

Participants Fifty-eight Australian specialist colleges and their subspecialty divisions.

Primary and secondary outcome measures Two reviewers extracted and coded research-related activities as learning outcomes, activities or assessments, by research stage (using, participating in or leading research) and competency based on Bloom’s taxonomy (remembering, understanding, applying, analysing, evaluating, creating). We coded learning and assessment activities by type (eg, formal research training, publication) and whether it was linked to a scholarly project. Requirements related to project supervisors’ research experience were noted.

Results Fifty-five of 58 Australian college subspecialty divisions had a scholarly project requirement. Only 11 required formal research training; two required an experienced research supervisor. Colleges emphasised a role for trainees in leading research in their learning outcomes and assessments, but not learning activities. Less emphasis was placed on using research, and almost no emphasis on participation. Most learning activities and assessments mapped to the ‘creating’ domain of Bloom’s taxonomy, whereas most learning outcomes mapped to the ‘evaluating’ domain. Overall, most research learning and assessment activities were related to leading a scholarly project.

Conclusions Australian specialist college research curricula appear to emphasise a role for trainees in leading research and producing research deliverables, but do not mandate formal research training and supervision by experienced researchers.

  • medical education & training
  • statistics & research methods
  • education & training (see medical education & training)
http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

View Full Text

Statistics from Altmetric.com

Strengths and limitations of this study

  • This is the first systematic exploration the constructive alignment of research related curricula of 58 subspecialty programmes of principle Australian specialist colleges.

  • We systematically explored the nature of Australian specialist colleges requirements regarding scholarly projects, project supervision and research training requirements.

  • We identified opportunities to enhance the development of trainee doctors’ research capabilities.

  • The scope of this study was confined to publicly available documents, and Australian colleges only.

  • Our analyses were focused on intended curricula rather than locally enacted curricula.

Background

In some countries, medical doctors undertaking their clinical specialty training (often referred to as registrars or trainees) are expected to do research during their specialist training.1–3 There are good reasons for encouraging research awareness and usage among trainees, with evidence suggesting that clinical care and patient outcomes are better in research-intense environments.4 5 The ways in which trainees can engage in research can be categorised in three domains: using, participating in or leading research6 (figure 1). The appropriate balance between these domains will vary according to the career plans of individual trainees. For aspiring physician-scientists it is appropriate for them to work towards a leadership role, while those planning a clinical career should focus more on learning how to translate research findings in clinical practice and policy development.

Figure 1

Glasziou's triangle (reproduced and adapted with permission from the Royal Australian College of General Practitioners) describes what clinician engagement in research should look like.6 This defines the research learning domains that were used to categorise learning activities described in documents published by Australian medical colleges.

In Australia, doctors undertake their clinical specialty training with accredited education providers (referred to as specialist colleges throughout this paper), typically commencing after a period of work as a junior medical officer in post-graduate years 4–5.7 Engaging trainees in these early specialist training years to improve research skills is difficult. A recent rapid realist review identified that effective training depends on a complex interplay between context, curriculum and pedagogic practices.8 It found that healthcare settings need to provide clinicians with the appropriate structure, guidance, support and mentorship from both college-set requirements and the healthcare institutions in which training occurs.8

As clinicians and researchers with responsibility for research capacity building in a university hospital, we observed trainee doctors submitting projects of wide-ranging methodological quality to our local human research ethics committee. Some described significant pressure to complete projects quickly to attain fellowships. Anecdotal evidence led us to believe that specialist college curricula requirements were driving trainee research behaviour and was an important target for systematic study.

Curriculum can be understood in three ways: the intended curriculum, enacted curriculum and experienced curriculum.9 10 The intended curriculum is the formal or written documentation describing the curriculum’s purpose and vision and includes learning objectives/outcomes and assessment.10 The enacted curriculum describes what happens when the curriculum is implemented whether it is in tutorials, working with others in the clinical setting or when the trainee studies at home, while the experienced curriculum focuses on the learners’ experience of the curriculum and what was learnt as a result. For this study, we have focused on the intended curriculum by exploring the research-relevant training outcomes and requirements documented by Australian specialist colleges.

We sought to systematically explore the intended research curriculum of Australian specialist colleges and its constructive alignment.11 Our goal was to understand how research training is embedded in specialist college curricula, the vision for trainees’ research development, the nature of scholarly project requirements and trainee project supervision. In doing so we hope to provide recommendations for research capability development opportunities for institutions that support and engage with trainees undertaking specialist training.

Methods

We conducted a content analysis12 of the intended research curriculum of Australian specialist colleges, including but not limited to medical, surgical and pathology colleges. Content analysis allows researchers to explore documents and make meaningful and replicable inferences from text.12 Through this process, key messages and concepts of texts can be garnered. This approach has been used in other research to understand the intentions of curricula.13

The intended curriculum we explored in this study included the stated learning aims and outcomes, learning activities, and methods of assessment. We selected the learning aims and outcomes as they communicate the educational intentions of learning programmes and are statements of what it is anticipated trainees will achieve.14 15 Moreover, the learning aims and outcomes tend to inform the decision making around learning activities and assessment. We also collected information about the learning activities trainees were to engage in and the methods of assessment.

We reviewed information on college websites that referred to trainee requirements. If documents referred to more than one curriculum due to updates (eg, trainees commencing before 2019 and trainees commencing 2019 onwards), only the most recent curriculum was included. Finally, to inform our analysis we drew on the concept of constructive alignment where it is argued that when alignment exists between learning outcomes, learning activities and assessment, trainees’ learn in a deep and meaningful way.16 A key principle of constructive alignment is that cohesion exists between what learners are expected to learn, the learning activities they engage in and the assessment used to determine learner progress.11

Publicly available information included curricula, handbooks, learning activities and assessment documents and webpages that could be accessed via college websites. Access to password-protected or hidden websites were not sought as it would be unreasonable for healthcare institutions and university departments that provide research support to trainee doctors to have access to this information. To prevent duplication of data, only individual specialist or sub-speciality college curricula were reviewed; joint training programmes were excluded.

Data extraction

Information related to traineeship research learning outcomes, learning activities and assessment activities were manually extracted into a pre-defined electronic data extraction form developed in Microsoft Access. We extracted sections of the curricula relevant to research only, including evidence-based practice. Basic college data were also extracted included name, abbreviation and duration of training.

To explore constructive alignment of the research cirrucula11 each relevant learning outcome, activity and assessment was coded separately, and information was extracted regarding the stage of research—namely using, participating or leading—6 and stage of competency, which was based on Bloom’s taxonomy—namely remembering, understanding, applying, analysing, evaluating, creating.17 If a learning outcome, learning activity or assessment referred to more than one level, then it was coded to the highest level as the assumption is that the previous levels are required. For example, a learning outcome that states that the trainee must ‘critically appraise articles and conduct research projects’ would be coded at the ‘leading research’ level, and as ‘creating’ in Bloom’s taxonomy.

Learning and assessment activities were also coded for the type of activity (eg, publication, protocol, conference attendance, journal club), whether the activity was mandatory, repeated (eg, attendance to more than one journal club) and linked to a scholarly project. Additional data collected about learning activities included whether the activity was a type of formal research training, and whether scholarly projects required a research-experienced project supervisor and a description of the scholarly project requirements.

Data were extracted from college websites by one of two authors (PF or IN) and all extracted data were reviewed by a third author (PS). Discrepancies were resolved by discussion, or by reference to a fourth author. The original study protocol (ethics application) can be viewed here: https://osf.io/aesnj/

Changes to original protocol

We initially planned to code specific skills involved for each learning outcome, activity and assessment, when in the training a requirement needed to be achieved, recognition of prior learning and study design allowed or required for scholarly projects. However, we abandoned this as descriptions were often too broad, ambiguous or missing all together.

We also found that a simple ‘mandatory: yes or no’ categorisation did not adequately represent the data on learning activities. They were re-categorised as:

  • Mandatory option: requirement had to be fulfilled to gain fellowship.

  • Option based system: requirement was one of several mandatory options where the trainee needed to fulfil at least one option to gain fellowship.

  • Points based system: requirement carried a pre-specified number of ‘points’ where trainees had to accumulate a certain number of ‘points’ to gain fellowship.

Analysis

Data were extracted from Microsoft Access and wrangled for college de-identification and preparation for analysis using Jupyter notebook in Python 318 environment with pandas,19 regex20 and numpy21 packages and the operator module. Data were analysed descriptively using Python 3 pandas,19 regex20 and numpy21 packages. Heatmaps were used to visualise constructive alignment between domains and were generated in R Studio22 using ggplot223 and egg24 packages. Interactive visualisations were also created in R using shiny,25 shiny themes25 and ggplot2.23 Python and R code and de-identified data are available here: https://osf.io/a6kr5/?view_only=13cf3220cb4e4fd1bede1275c9de34d3

Results

We analysed documents from 58 Australian specialist colleges and their subspecialty divisions. These included all sections of the Royal Australasian College of Physicians, the Royal Australasian College of Surgeons, Royal College of Pathologists of Australasia, the Royal Australian College of General Practice and several other smaller specialist colleges.

Constructive alignment

Colleges are enumerated but not identified by name in the vertical axis of figures 2 and 3. Higher frequency counts for documents that mapped to the relevant domains are represented as darker shades in the heat map. The legend to figures 2 and 3 provides numeric calibration. The colleges differed in documentary detail, hence vertical (between-college) comparisons will be affected by this. Horizontal (within-college) comparisons are likely to have greater validity.

Figure 2

Constructive alignment between learning outcomes, learning activities and assessment activities across research levels—using, participating in and leading research—including those affiliated with scholarly projects. To view without scholarly project requirements, see the interactive visualisation available here: https://p-stehlik.shinyapps.io/registrar_interactive_vis/

Figure 3

Constructive alignment between learning outcomes, learning activities and assessment activities across Bloom’s taxonomy, including those affiliated with scholarly projects. To view without scholarly project requirements, see the interactive visualisation available here: https://p-stehlik.shinyapps.io/registrar_interactive_vis/

While the learning outcomes in the ‘using research’ domain had the highest frequency counts (denoted by the most darkly shaded boxes in figure 2), this was not matched by learning activities and assessments in that domain which were sparser. The most uniformly shaded domain was ‘leading research’, where frequency counts generally were lower for learning activities than for learning outcomes or assessments. In this domain, many of the learning and assessment activities were related to the completion and reporting of projects (figure 2). The most sparsely populated domain was ‘participating in research’, with low frequencies in the learning outcomes column and almost nothing in the other columns.

The mismatch in constructive alignment is also apparent when requirements are visualised using Bloom’s taxonomy, with learning outcomes most shaded in the ‘evaluating’ domain, yet learning activities and assessments most uniform shading across the ‘Creating’ domain (figure 3).

Learning and assessment activities

Of the 58 colleges reviewed, 56 required trainees to complete some form of scholarly project, with 51 requiring all trainees to complete a project, one requiring a scholarly project through an options-based system, and three through a points-based system. One college offered scholarly project as an optional requirement and two did not require the trainees to conduct any form of scholarly project (table 1). On the other hand, only two colleges stipulated that the project supervisor had to be an experienced researcher, 30 encouraged it and 24 did not mention supervisor research experience. Of these, 12 stipulated project supervisor requirements (eg, college fellow) but did not mention research experience, nine did not provide any description of the characteristics of the project supervisor, three did not mention a project supervisor anywhere in the publicly available documents (table 1).

Table 1

Research training requirement

Eleven colleges required some form of formal research training. Thirty-one colleges stated that research training was optional, and it was not mentioned by 16 colleges (table 1). Most formal research training was in the form of online modules, or workshops (eg, 2-day workshop) and pre-approved seminars, and some required webinars and courses. Other research-related learning activities included conference and journal club attendance, and some form of critical appraisal exercises (eg, leading journal club, critical appraisal tasks, topic review, etc).

Of the 56 colleges that required or offered the option of a project, 47 required the submission of a thesis style report (6, options-based; 3, points-based), 19 required publication as first or second author in order to gain fellowship (10, options-based; 5, points-based) and 20 required a conference presentation (5, options-based; 7, points-based) (table 1). Other research-related assessment activities included written examination, supervisor reports, written reflections on learning and oral presentation or written report of critical appraisal tasks.

Overall, most of the research-related learning activities and assessments were affiliated with the completion of a scholarly project (to view figure 2 without scholarly project requirements, see the interactive visualisation available here: https://p-stehlik.shinyapps.io/registrar_interactive_vis/).

Discussion

Based on our analysis of their publicly available documents, there is a lack of constructive alignment of the research curricula prescribed by the Australian specialist colleges, inadequate supervision and an absence of appropriate training.

Lack of constructive alignment between learning outcomes, activities and assessment risks meaningful learning about research. For example, the approach to learning about research may be superficial and focused on the tasks at hand rather than generating deep understanding.26 This may not be unique to research curricula; however, exploring the constructive alignment of the broader range of college curricula was outside the scope of this work.

Our analysis suggests Australian colleges place relatively little emphasis on acquiring research skills, obtaining experience through participation in well-run projects, and being supervised by experienced researchers. In contrast, most colleges appear to give weight to completion of research projects, with some emphasising authorship and presenter status. Indeed, most of the learning and assessment of research skills are affiliated with trainee-lead scholarly projects. It is unclear whether this is appropriate for all trainees; however, it makes research productivity important for career progression, and this continues through all stages of specialist training.27 This is despite the fact that relatively few trainees to go on to hold career positions that are substantially concerned with the conduct of research.28

These findings highlight the need to review research training curricula, but they also point to the possible need for a cultural re-alignment. Some of the publicly documented college requirements encourage the view that specialist trainees will naturally take on leadership roles in research without having acquired the necessary skills. This view may have its origins in historical approaches to clinical research, typically led by specialist clinicians who may have had little formal training in research methods. The current consensus favours research of important topics by multi-disciplinary research teams, in which specialist doctors (trainee or otherwise) have an essential but not necessarily leading role.29–32 In our view, research training curricula need to reflect this, emphasising the importance of acquiring a range of skills and gaining experience of teamwork. There is evidence that meaningful research learning experiences can enhance trainees’ enthusiasm and confidence for research especially if they successfully complete the research.33 34 This is most likely under the supervision of a trained researcher.8 We found that many colleges require that the supervisor is a college fellow, but this does not guarantee possession of the necessary research-related skills.29

Lack of constructive alignment, inadequate supervision and a lack of appropriate training is counter-productive and may lead to hurried poor quality work.35 Our local trainees have described their feelings of lack of preparedness and the focus on project completion rather than participating in and learning from well-run and productive research teams.36 Specialist trainees are not unique in this regard; these pressures have been described in the earlier phases of clinical training, including by medical students.37 38 At each stage of training, the problem seems to be a focus on individual projects. There may be major advantages in encouraging trainees to form research collaboratives39 or research coursework pathways.40 These have been shown to be productive, reduce pressure on individuals and may provide a good vehicle for introducing research methods training.39 40

An additional concern is that college curricula may inadvertently contribute to research waste. In 2014 a series of papers published by The Lancet41 framed the causes of research waste as conforming to the different stages of research projects. These include failure to take adequate account of previous studies when planning new work, poorly formulated research questions, inadequate study designs, inefficient study conduct, incorrect analyses and failure to report any or all results. The Lancet series conclusions were primarily aimed at funders and publishers of research; however, the large number of research projects undertaken by college trainees make them worthy of closer attention. While our analyses did not look at the quality of trainees’ research studies or reports, curricula characteristics that we have highlighted (lack of training, emphasis on project completion, inexperienced supervisors) may create the conditions that lead to research waste. Our interactions with trainees also lead us to believe that our concerns are well founded—our future work aims to quantify this.

Without the right support systems in place, mandated trainee-led scholarly projects may not be contributing to meaningful research development. More importantly, they could be at risk of contributing to research wastage, be a burden to supervising clinicians, and may even pose harm to patients. In the words of the late Doug Altman, “carrying out a sensible study, even on a small scale, is indeed useful, but carrying out an ill designed study in ignorance of scientific principles and getting it published surely teaches several undesirable lessons.”42

Limitations

To our knowledge, this is the first study to systematically explore the nature of Australian specialist college research curricula, focusing on constructive alignment, the nature of college mandated scholarly project, project supervision and formal research training requirements.

However, we recognise several limitations in this work. Most importantly, in this report we have taken a high-level view of our data and have made several generalisations. Our analyses were confined to intended rather than enacted and experienced curricula, as described in the publicly available documents. We also did not look at the nature of research products that result from college-mandated activities. Further research will assess the quality of the projects and publications arising from these training programmes, and the subjective experiences of the trainees.

While all the reviewed colleges except two had publicly available trainee curricula and handbooks, some of the college learning activity objectives and assessment criteria were not publicly available. It is possible that documents that we could not access address some of our concerns. However, the stated specialty college research requirements remain at odds with good research practice. Some colleges have already begun to question the value of this kind of research training.43 44 A fundamental overhaul of educational approach and emphasis in the research curriculum may be required.

While most of the college curricula also apply to trainees in New Zealand, we cannot say whether the data presented here are generalisable to specialty training organisations in other countries. However, similar criticisms have been made of research training in the UK.27 The underlying problems of poor reproducibility and waste in research are widespread and clinical specialty training organisations around the world need to consider whether their requirements are contributing to this.

Conclusion

The intended research curricula of Australian specialty colleges seem to lack constructive alignment and emphasise a role for trainees in leading research, rather than using or participating in research. Most of the research learning and assessment activities are affiliated with trainee-led scholarly projects even though most colleges do not require a research-experienced supervisor for these projects, nor does there appear to be adequate formal research methods training embedded in specialty college curricula. The focus on leading research that, on paper, is unaccompanied by adequate supervision and appropriate training are at odds with contemporary efforts to mitigate the growing problem of poor reproducibility and wastage in medical research. Future work will aim to quantify the outcomes of enacted curricula, including quality of research outputs, and capture the subjective experiences of doctors undertaking these requirements.

References

View Abstract

Footnotes

  • Twitter @chnoble

  • Contributors Conceptualisation: PS, CB, CN; Data curation: PS; Formal analysis: PS; Funding acquisition: PS; Investigation: PS, PF, IN, DH, PG, CB, CN; Methodology: PS, CB, CN, DH, PG; Project administration: PS; Supervision: PS; Validation: PS; Visualisation: PS; Writing–original draft: PS, DH, PG; Writing–review and editing: : PS, CB, CN, DH, PG, PF, IN. PS is the guarantor of the article.

  • Funding This work was funded by internal departmental funds from Bond University and Gold Coast Health.

  • Disclaimer PS, DH and PG are responsible for providing teaching in Evidence-Based Practice to staff at Gold Coast Health (GCH). CB is advanced research development officer and at the time of this study CN was a principal medical education officer and principal research fellow (Allied Health) at GCH. This project arose from discussions between the authors about how to align Evidence-Based Practice teaching and research development/support with both the needs of clinicians and the requirements of the responsible specialist colleges. Discussions with trainees indicated that they felt pressure to complete projects without adequate training, time or resources.

  • Competing interests None declared.

  • Patient consent for publication Not required.

  • Ethics approval This study received an ethics waiver from the Gold Coast Hospital and Health Service Human Research Ethics Committee (LNR/2018/QGC/46317).

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data availability statement Data are available in a public, open access repository. Python and R code and de-identified data are available on OSF: https://osf.io/a6kr5/?view_only=13cf3220cb4e4fd1bede1275c9de34d3.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.