Article Text

Download PDFPDF

Protocol
Study protocol: content and perceived educational utility of different modalities of clinical teaching visit (CTV) workplace-based assessments within Australian general practice vocational training: a cross-sectional study
  1. Alison Fielding1,2,
  2. Benjamin Eric Mundy2,
  3. Amanda Tapley1,2,
  4. Linda Klein1,2,
  5. Sarah Gani3,
  6. Michael Bentley4,
  7. Rachael Boland4,
  8. Lina Zbaidi5,
  9. Mieke L van Driel6,
  10. Elizabeth Holliday1,
  11. Parker Magin1,2
  1. 1School of Medicine and Public Health, Faculty of Health and Medicine, The University of Newcastle, Callaghan, New South Wales, Australia
  2. 2NSW & ACT Research and Evaluation Unit, GP Synergy, Regional Training Organisation, Mayfield West, New South Wales, Australia
  3. 3Medical Education, GP Synergy, Regional Training Organisation, Sydney, New South Wales, Australia
  4. 4General Practice Training Tasmania, Hobart, Tasmania, Australia
  5. 5Northern Territory General Practice Education, Darwin, Northern Territory, Australia
  6. 6Primary Care Clinical Unit, The University of Queensland Faculty of Medicine, Herston, Queensland, Australia
  1. Correspondence to Dr Alison Fielding; alison.a.fielding{at}newcastle.edu.au

Abstract

Introduction Clinical teaching visits (CTVs) are formative workplace-based assessments that involve a senior general practitioner (GP) observing a clinical practice session of a general practice registrar (specialist vocational GP trainee). These visits constitute a key part of Australian GP training. Despite being mandatory and resource-intensive, there is a paucity of evidence regarding the content and educational utility of CTVs. This study aims to establish the content and educational utility of CTVs across varying practice settings within Australia, as perceived by registrars and their assessors (‘CT visitors’). In addition, this study aims to establish registrar, CT visitor and practice factors associated with CTV content and perceived CTV utility ratings.

Methods and analysis This study will collect data prospectively using online questionnaires completed soon after incident CTVs. Participants will be registrars and CT visitors of CTVs conducted from March 2020 to January 2021. The setting is three Regional Training Organisations across four Australian states and territories (encompassing 37% of Australian GP registrars).

Outcome factors will be a number of specified CTV content elements occurring during the CTV as well as participants’ perceptions of CTV utility, which will be analysed using univariate and multivariable regression.

Ethics and dissemination Ethics approval has been granted by the University of Newcastle Human Research Ethics Committee, approval number H-2020-0037. Study findings are planned to be disseminated via conference presentation, peer-reviewed journals, educational practice translational workshops and the GP Synergy research subwebsite.

  • education & training (see medical education & training)
  • primary care
  • medical education & training
http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Strengths and limitations of this study

  • This study will sample from a diverse range of general practice training settings, including all urban and rural classifications.

  • Although the COVID-19 pandemic has impacted the amount of face-to-face clinical teaching visits (CTVs) that will occur during the study period, this has provided an opportunity to gather evidence on different CTV modalities and remote CTV options.

  • A limitation of this study is that outcomes will be self-reported rather than observations of participant behaviour.

Introduction

General practice training

In Australia, and in many other countries, specialist vocational general practice (family medicine) training occurs via an apprenticeship-like model.1 In Australia, the training of vocational general practice/family medicine trainees (known as general practitioner ‘(GP) registrars’) is overseen by the Royal Australian College of General Practitioners (RACGP) and the Australian College of Rural and Remote Medicine (ACRRM). Regional Training Organisations (RTOs)—geographically defined not-for-profit educational organisations—are tasked with delivering regionalised training and education, tailored to meet regional needs. The two colleges conduct summative licensing examinations and RTOs conduct a range of formative assessments, including in-training assessments.

Summative licensing assessments ensure that a registrar’s competencies have developed to a satisfactory, criterion-based standard. While formative assessments also examine a registrar’s competencies, the main objective is to provide actionable feedback to further their professional development across the course of their training.2 While both methods of assessment are essential components of medical training, increasing consideration is being placed on programmatic assessment, in which formative assessments play an integral part. Due to the timing of their delivery, formative assessments have the potential to both identify the need and provide the opportunity for tailored development via the delivery of feedback.3

Workplace-based assessments

A key mode of formative assessment undertaken by medical trainees, including GP registrars, is workplace-based assessment (WBA). During a WBA, a trainee’s competencies are assessed at the apex of Miller’s pyramid (does),4 by an assessor observing the trainee in practice. Ultimately, the goal of a WBA is to determine potential areas of improvement for the trainee and provide feedback that could be translated into practice. The psychometric properties of WBAs used in medical training have been substantiated by considerable research involving psychometric evaluations.5–9

Though the reliability and validity of a number of WBAs have been established, a WBA can sometimes have limitations in practice. For example, a recent study from Ireland found 20% of GP trainees and trainers perceived WBAs to be a single use, ‘box-ticking’ exercise.10 This suggests that, for a proportion of those participating, WBAs were used suboptimally. Contrary to this perception as a ‘tick-box’ exercise, WBAs must be part of an educational process. Holmboe et al2 remarks that competency-based medical education needs to continually link assessment of competency and the provision of feedback. Reoccurring assessment provides multiple opportunities for feedback, and, therefore ongoing reflection and development.

Other characteristics are important to ensure that the feedback is effective. For example, when feedback is too vague, it can inhibit learning.11 Furthermore, the literature suggests that, in addition to the provision of specific feedback, it is important to provide the trainee with an opportunity to reflect on their performance.12 Observational evidence involving GP trainees has found that the rate of translating feedback into action is higher if the feedback is specific (compared with nonspecific feedback), and higher again if the specific feedback is coupled with trainee reflection.13

While WBAs are, by definition, based within actual practice, there is only limited evidence for the utility of WBAs in affecting clinicians’ training and, particularly, their subsequent clinical performance.14 15 Most evaluations of WBA utility are conducted at lower levels of Barr’s evaluation hierarchy (learner’s reactions; modification of attitudes and perceptions; acquisition of knowledge and skills) rather than at higher levels (change in behaviour; change in organisational practice; benefit to patients).16

Clinical teaching visits

Clinical teaching visits (CTVs) are a particular WBA used throughout Australian vocational GP training. CTVs involve a trained and experienced GP (external to the practice), assigned by their respective RTO, assessing a registrar’s competencies.17–20 During a visit, of approximately 3-hour duration, the CT visitor observes the registrar’s consultations and engages in feedback-oriented discussion.21 22 Such discussion is intended to address both strengths and areas for improvement for the registrar, aiming to facilitate their professional development. While a CTV in itself is considered a WBA, due to the length of the visit, other forms of WBA can take place within a CTV. For example, a mini clinical evaluation exercise (mini-CEX) or random case analysis (RCA) can be nested within a CTV. As such, the assessment of the registrar’s competence and the feedback provided can be guided by various tools during the visit.

CTVs are mandated nationally by the RACGP, with GP registrars required to complete a minimum of five CTVs during their training period.23 ACRRM requires nine mini-CEX to be completed during training. These mini-CEXs are conducted within a CTV-like format.24

Educational content and utility of CTVs

Despite the central, mandated place of CTVs within Australian GP vocational training and their resource-intensive nature, there is a paucity of empirical evidence supporting their use. In fact, most of the evidence that exists concerning CTVs relates to the various WBAs that can take place within them. For example, the mini-CEX, a tool with established content and construct validity,5 9 25 26 can be used during a CTV. However, to our knowledge, the mini-CEX has not been empirically studied when used within a CTV setting. Regarding discrete tools that have been studied within a CTV setting, there is evidence to support the addition of RCA during a CTV. This evidence comes from a study involving qualitative analysis of registrar and assessor feedback on an RCA tool.21 27 While it demonstrated support for the tool’s use during CTVs, the frequency of its use is currently unknown.

There is also some evidence for structured feedback forms such as Hays’ Consultation Assessment Scale (CAS). Hays’ CAS has been shown to demonstrate validity and interrater reliability via evaluations of videotaped sessions28 and can be used to guide observation and registrar feedback. A small study of 21 CTVs investigating the use of Hays’ CAS during the CTV29 reported that registrars found feedback administered via this form useful for guiding further discussions with their medical educator (that is, useful at the lowest level of Barr’s evaluation model).16

GP Synergy, the RTO based in New South Wales (NSW) and the Australian Capital Territory (ACT), currently uses the General Practice Registrar Competency Assessment Grid (GPR-CAG) for assessment of registrar’s performance across the entirety of a CTV. The GPR-CAG has demonstrated initial content and construct validity.30 However, the perceived educational utility of the GPR-CAG itself as a CTV assessment instrument has not been evaluated.

The above studies of CTVs have focused on a single instrument (the RCA)21 27 incorporated within a portion of a CTV, evaluated the utility of an instrument for providing feedback following a CTV (CAS)29 or have derived and evaluated the psychometric properties of an instrument for use in evaluating registrar performance during a CTV (GPR-CAG).30 Also, the first two of these three studies were small in terms of sample size as well as geographical coverage. Thus, there is limited empirical literature concerning CTVs and no studies have assessed the utility of the CTV as a whole, particularly at the higher levels of Barr’s evaluation model. Additionally, the variability of CTV content and utility across different practice or training contexts are also largely undocumented.

There is some evidence from small-scale local evaluations conducted within RTOs to indicate that registrars and supervisors perceive CTVs to be a useful and important part of registrar training (Personal communication with: Christopher Starling 2012; Karen Flegg 2008), but these do not cover the views of the CT visitors and, to our knowledge, there is no peer-reviewed published evidence on this topic. This is a key evidence gap for a tool that is so broadly used and deeply integrated into Australian GP education.

CTVs in different Australian training settings

Frameworks are in place within each RTO to guide the way CTVs are conducted,17–20 however, CT visitor training can be ‘siloed’ between RTOs, meaning there is little coordination between RTOs regarding the guidance provided to CT visitors.31 Consequently, the way CTVs are conducted across different regional and practice settings can vary. This may be appropriate as it is unlikely an entirely standardised CTV would be able to adequately account for practice, cultural and socioeconomic variability. Factors such as patient demographics, practice physical environment and organisation, local health resources and local health referral pathways have the potential to affect performance on specific assessments.32–34 It is important to critically assess the similarities and differences of CTV elements across different training settings, taking into account these contextual factors.

Establishing the content and utility of CTVs in GP vocational training is of vital importance for informing best practice regarding GP training. In this study, we aim to establish the content and perceived educational utility of CTVs in three geographically and demographically distinct general practice vocational training programmes.

The objectives of the study are to establish:

  1. The clinical and educational content of CTVs in three Australia RTOs.

  2. Perceived educational utility of CTVs from the perspective of registrars and of CT visitors in three RTOs.

  3. Registrar, CT visitor and practice factors associated with CTV content.

  4. Registrar, CT visitor and practice factors associated with perceived CTV utility ratings.

  5. Agreement of registrars’ and CT visitors’ reports of CTV content and of registrars’ and CT visitors’ ratings of CTV utility.

Methods and analysis

Study design

This observational study will collect data prospectively using online questionnaires completed soon after incident CTVs. Though participants may contribute data on multiple occasions, analysis will be cross-sectional (adjusting for clustering of CTVs within registrars or CT visitors).

Study setting

The study will include registrars and external CT visitors from three RTOs—GP Synergy, General Practice Training Tasmania (GPTT) and Northern Territory General Practice Education (NTGPE). GP Synergy is the largest of these RTOs, with an intake of approximately 500 registrars per year. Across all Australian RTOs, GP Synergy trains 32% of Australian registrars in general practice terms, covering both NSW and ACT. GPTT and NTGPE provide training to all registrars in Tasmania and the Northern Territory, respectively, which constitutes 5.2% (2.6% GPTT and 2.6% NTGPE) of registrars in general practice terms. These three RTOs cover a wide range of practice and demographics including urban, rural, remote and diverse socioeconomic settings. This will allow us to capture the variability of CTVs conducted throughout different Australian GP training settings.

Eligibility criteria

All registrars from GP Synergy, NTGPE and GPTT who are in general practice terms during the study period (March 2020–January 2021) will be eligible to participate after each of their CTVs. The external CT visitors who conduct these CTVs will also be eligible.

Immediately prior to the commencement of data collection for the study, the onset of the COVID-19 pandemic led to substantive changes to CTV delivery modalities. The previously practised face-to-face CTVs (as described in the introduction) were no longer practicable, and alternative CTV modalities were adopted. The three RTOs used variations of CTV modalities (see table 1). Participants involved with these assessments will be invited to participate, regardless of the CTV modality.

Table 1

Alternative CTV modalities used during the peak COVID-19 pandemic period

Patient and public involvement

No patients have or will participate in the design, implementation or interpretation of the study.

Recruitment

All registrars and external CT visitors will be invited via email by their respective RTOs to participate in the online questionnaire about each of their CTVs during the study period, within 2 days of the index CTV taking place. The study period is scheduled to be March 2020–January 2021. Due to the unforeseen impact of the COVID-19 pandemic on the scheduling of routine CTVs, data collection may continue beyond this period as required, to ensure adequate power is obtained for analysis. Completion of the questionnaire within 2 days of the CTV occurring will limit recall bias. As each CTV is different, participants who have completed more than one CTV during the study period will be invited to participate after each CTV they complete. Contact information held by each RTO will be used to distribute email invitations to participate. Contact information will not be shared between RTOs, in order to maintain confidentiality. Completion of the questionnaire will be voluntary for participants and no reward (financial or otherwise) will be offered for completing the questionnaire.

Data collection and management

Separate online registrar and CT visitor questionnaires were piloted in GP Synergy’s Hunter Manning Central Coast region during 2019. Registrar and CT visitor questionnaires took an average of 3 and 5 min, respectively, to complete. The questionnaires used in the current study are iterations of the pilot questionnaires. Questionnaire development was influenced by the analysis of the pilot data and further consultation with the project team, which included experienced GPs, GP registrars and GP educators (including CT visitors).

Each RTO will be responsible for collecting and managing data from participants within their respective regional settings, to maintain participant anonymity. This will entail the creation of a master list that will contain registrar and CT visitor contact and CTV schedule information, registrar practice characteristics as well as unique registrar, CT visitor and CTV identifiers. This information will be updated weekly to maintain accurate CTV scheduling information and will be used to distribute invitations, within 2 days of each visit, via Survey Monkey using its collector methodology. Survey Monkey’s collectors can be used to import data, such as contact information, from excel files into email invitations. In addition, the use of collectors allows for the association of CTV and participant identifiers with response data. The use of this method enables the establishment of registrar–CT visitor dyads for analysis of registrar–CT visitor agreement and for our analyses to be adjusted for clustering of CTVs within registrars or CT visitors.

Outcomes

As this is an exploratory study of an under-researched area, there will be a large number of descriptive outcomes. There will be three primary outcomes elicited from both registrars and CT visitors and two primary outcomes elicited from registrars only. Multivariable analyses will be conducted for these eight outcomes (for a full overview of outcomes, see table 2).

Table 2

Outcomes elicited from registrars and/or CT visitors, whether prevalence will be calculated, and whether multivariable analysis will be performed with that outcome

Some outcomes may also be used as covariates in other multivariable analyses. Additional independent variables are seen in table 3. Outcomes will encompass the occurrence of a range of specific clinical and educational elements within the CTV, plus a rating of the utility of the CTV, and ratings of the registrar’s likelihood of changing their practice, and of changing their learning approach, as a result of the CTV.

Table 3

Independent variables

Table 4

Specific discussion topics listed in the questionnaire

Sample size

The pilot study of 175 CTVs demonstrated response rates of 49% and 53% for registrars and CT visitors, respectively. Using this information, with an estimated 1169 visits occurring within the study period, we estimate that the current study will yield a sample size of 561 individual CTV registrar responses and 643 individual CTV visitor responses. We estimate that there will be approximately 350 CTV registrar–CTV visitor dyads available for analysis. This will allow estimation of the prevalence of individual CTV elements, with 95% CIs, to within an error margin (half the CI width) of 3%–4%. The anticipated response rates are considered acceptable for cross-sectional study with GP participants.35

Statistical methods

The prevalence of categorical outcomes will be assessed as proportions with a 95% CI. Continuous outcomes will be assessed using means and SD with a 95% CI.

For the eight outcomes nominated in table 2 for multivariable analysis, the frequencies of categorical independent variables will be compared between outcome categories using χ2 tests for all variables, except when Fisher’s exact test will be used (due to an expected count of less than 5 in 25% or more cells). For continuous independent variables, means will be compared using a t-test or nonparametric equivalent if appropriate.

Univariate and multivariable associations of the eight outcomes nominated in table 2 will be tested using logistic regression within the generalised estimating equations framework to account for repeated measures within registrars and visitors. An exchangeable working correlation structure will be assumed.

Registrar–CT visitor agreement will be established using Cohen’s κ.

Ethics and dissemination

Consent or assent

The completion of the questionnaire by the participant will be considered implied consent.

Confidentiality

Each RTO will be responsible for downloading data from SurveyMonkey and deidentifying it before it is sent to GP Synergy for analysis. Master lists were held by each RTO and will be stored in password-protected files. Separate password-protected files will be used to store data and all complete, deidentified questionnaire data will be held and only be accessible by the study research team at GP Synergy.

Access to data

Data will only be accessible by investigators and statisticians committed to the project.

Dissemination policy

Study findings will be disseminated via presentations at general practice and medical education conferences as well as publication in peer-reviewed journals. GP Synergy will also disseminate findings via workshops run for GP Synergy stakeholders such as medical educators, CT visitors and supervisors. Each RTO will distribute the findings of the study to the participants within their respective domains. Thorough documentation of results and study methodology will be made available to interested parties on GP Synergy Research subwebsite: https://research.gpsynergy.com.au/. Publications will be written by the authoring team without the use of professional writers.

Data availability

Nonidentifiable data may be shared with other parties on reasonable request to encourage scientific scrutiny and to contribute to further research and public knowledge.

Discussion

Anecdotally, registrars and CT visitors acknowledge CTVs as a highly valuable educational experience (Personal communication with: Christopher Starling 2012; Karen Flegg 2008). However, there is little published evidence regarding the content of CTVs. While a small number of studies have examined the use of particular tools in a CTV setting,21 27 the overall content of CTVs as a whole has not been systematically documented. Similarly, there is little published evidence examining the educational utility of CTVs.29 This is a key evidence gap for an educational element that is so resource demanding and mandatory for all Australian registrars. As different regional settings can call for tailored variation to elements of training,32–34 it is important to systematically document the content of CTVs and the relative value they hold for registrars and CT visitors, as this information will help inform a best practice approach to GP training in varying settings throughout Australia.

The aim of this study is to address the current evidence gap regarding CTVs by eliciting registrar and CT visitor perceptions of CTV content and educational utility in varying practice settings via an online questionnaire. As our sampling strategy is broad and aims to include registrars and CT visitors from all urban and rural classifications, we expect the results to be generalisable to all Australian RTOs. The results may also be generalisable to other countries using direct observations in an apprenticeship-type model for GP training.

The findings of this study will be of importance in the domain of medical education, particularly regarding competency-based medical education in Australia. Demonstrating the prevalence of content elements and utility ratings as well as the corresponding associations between these outcomes across varying training settings could inform and influence the guidance RTOs provide to CT visitors regarding conducting visits. As a result, RTOs and CT visitors may be able to tailor the educational experience the registrar receives based on the needs imposed by the regional setting.

Limitations

A major limitation of this study involves the suspension of face-to-face CTVs in response to the COVID-19 pandemic, resulting in data being collected for a smaller number of face-to-face CTVs so far. At the time of writing, it remains unclear how long face-to-face CTVs will remain suspended. Although each RTO has implemented alternative CTV modalities, these differ from face-to-face CTVs and have not yet become well established. This, however, has provided us with a unique opportunity to collect evidence on a range of CTV modalities, which may inform how CTVs can be conducted in the future. This is of particular relevance for the ongoing development of effective yet resource-efficient alternatives to traditional, face-to-face WBAs.

Another limitation of this study is the subjective nature of determining the educational utility of CTVs. While our study includes assessments at a higher level of Barr’s evaluation model16 (level 3: change in behaviour) rather than purely lower levels (learners’ reactions and changes in attitudes, knowledge and skills) as studied in much previous research, we will employ self-reported intent as a surrogate measure of change rather than direct measures of behaviours. A self-reported measure, however, was deemed most appropriate as deriving an objective measure for the educational utility of CTVs was beyond the scope and resource capacity of this study. Also, an observational rather than experimental design was chosen. Implementing randomised control trials with CTVs as an intervention is not feasible, as CTVs are a mandatory element of GP training in Australia.19 24

Conclusion

This study will contribute valuable evidence regarding the content and perceived educational utility of CTVs within Australian vocational general practice training. This will help to inform the ongoing optimisation of CTV delivery in Australia and may help inform how other WBAs are conducted in similar apprenticeship-like vocational medical training programmes internationally.

References

Footnotes

  • Contributors AF, AT, LK and PM conceived the CTV study. AF, AT, BEM, EH, LK, LZ, MB, MLvD, PM, RB, SG substantially contributed to the design of the study reported in this protocol. AF, AT, BEM, PM drafted this paper. This paper includes intellectual content of all authors, and all authors have provided final approval of the current version for publication.

  • Funding This research project is supported by the Royal Australian College of General Practitioners with funding from the Australian General Practice Training Programme: An Australian Government initiative (grant number ERG2020-00002). In-kind support will be provided by GP Synergy, GPTT and NTGPE.

  • Competing interests AF, AT, BEM, LK, PM and SG are employees of GP Synergy. MB and RB are employees of GPTT. LZ is an employee of NTGPE. PM is a BMJ Open Editorial Advisory Board member.

  • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.