Does current provision of undergraduate education prepare UK medical students in ENT? A systematic literature review
====================================================================================================================

* Gary R Ferguson
* Irina A Bacila
* Meenakshi Swamy

## Abstract

**Objective** To systematically identify and analyse all published literature relating to the provision of undergraduate education for preparedness in ear, nose and throat (ENT) surgery, as perceived by medical students and clinicians in the UK.

**Design** Systematic literature review.

**Data sources** 5 major databases were searched: MEDLINE, EMBASE, ERIC, Cochrane and Web of Science. The literature search was conducted from February to April 2015.

**Eligibility criteria for selecting studies** Primary research or studies that report on the provision of undergraduate education for preparedness in ENT, from the perspective of medical students and clinicians in the UK. The timescale of searches was limited from 1999 onwards (ie, the past 15 years).

**Data extraction** The literature search was conducted by 2 independent reviewers. Search terms used involved the combination and variation of 5 key concepts, namely: medical student, clinician, ENT, undergraduate medical education and UK. A data extraction form was designed for and used in this study, based on guidelines provided by the UK National Health Service (NHS) Centre for Reviews and Dissemination. Textual narrative synthesis was used for data analysis.

**Results** A total of 7 studies were included in the final review. 4 main themes were identified: confidence in managing patients, teaching delivery, student assessment and duration of rotations. A consistent finding in this review was that the majority of final year medical students and junior doctors did not feel adequately prepared to practise ENT. Important factors influencing preparedness in ENT included the duration of clinical rotations, the opportunity for hands-on learning and formal assessment.

**Conclusions** The findings of this review suggest the need for further development of the ENT undergraduate curricula across the UK. However, there is insufficient evidence from which to draw strong conclusions; this in itself is beneficial as it highlights a gap in the existing literature and supports the need for primary research.

*   ENT
*   OTOLARYNGOLOGY
*   UK Medical Schools
*   Undergraduate Medical Education
*   Medical Students
*   Clinicians

### Strengths and limitations of this study

*   A comprehensive literature search was carried out using five major databases.

*   The review involved a diverse range of participants.

*   ‘Preparedness’ in this review was more consistent with the level of confidence, rather than a prediction of performance.

*   Only seven studies qualified for inclusion in this review.

*   The use of questionnaires to collect data in each study limited the opportunity for participants to express their perspectives.

## Introduction

Ear, nose and throat (ENT) is the third largest surgical specialty with related problems frequently encountered in a range of other disciplines including general practice and emergency medicine.1–4 In general practice, for example, ENT accounts for 25% of adult and 50% of all paediatric consultations.2 ,4 Despite this, ENT forms only a small part of the undergraduate medical programme. According to the General Medical Council recommendations, the curriculum must help students to develop the knowledge, skills and behaviour needed for clinical practice.5 ,6 Students and clinicians have raised concerns about the provision of undergraduate education for preparedness in ENT.7–13 These concerns relate to a lack of confidence among final year medical students and junior doctors in dealing with common ENT problems.7–13

Most of the literature relating to the provision of undergraduate education for preparedness in clinical practice has focused on the general preparedness of junior doctors for starting work.14–19 A national survey found that over 40% of junior doctors in the UK did not feel prepared for their first post.14 A follow-up survey showed improvement in the level of preparedness, but revealed wide variation across UK medical schools.15 A lack of preparedness has also been reported internationally, with medical students in America and Australia feeling unprepared for clinical practice.20–22

Illing *et al*17 in 2013 conducted a qualitative study involving 60 graduates from three UK medical schools with different curricula to determine whether they were prepared for clinical practice. Junior doctors felt prepared in terms of communication skills and team working, but less prepared for being on-call, prescribing, managing acutely unwell patients and dealing with paperwork. This highlighted the importance of medical students having hands-on learning experience in a supervised clinical setting.

Matheson and Matheson18 in 2009 captured the views of consultants and specialist registrars working alongside junior doctors in two UK teaching hospitals. This study found that junior doctors were best prepared for communicating and asking seniors for help but less prepared in terms of clinical and practical skills, suggesting the need for further training at undergraduate level.

Tallentire *et al*19 in 2011 conducted a systematic literature review to investigate the perceived preparedness of UK medical graduates in acute care. Acute care was identified as an area of weakness by medical graduates and senior clinicians. This study highlighted the need for medical students to gain more experience in managing acutely unwell patients during their undergraduate training.

Few studies have considered the provision of undergraduate education for preparedness in ENT.7–13 A lack of specialty-specific understanding leaves ENT undergraduate programmes without clear guidance on how best to prepare students for clinical practice.8 ,12 This systematic review of the existing literature will form a starting point to better understand the provision of undergraduate education for preparedness in ENT, as perceived by medical students and clinicians in the UK. It is intended that the findings of this review will contribute to further development of the ENT undergraduate curricula to increase the preparedness of newly qualified doctors for clinical ENT.

### Research question

How is the provision of undergraduate education for preparedness in ENT perceived by medical students and clinicians in the UK?

*Objective*: To systematically identify and analyse all informative, published literature relating to the provision of undergraduate education for preparedness in ENT in the UK.

## Methods

A systematic literature review was utilised in this study. Five major databases were searched: MEDLINE, EMBASE, ERIC, Cochrane and Web of Science. The search terms used were variations of five key concepts, namely: medical student, clinician, ENT, undergraduate medical education and UK (see table 1). The reference lists of identified literature were also screened for identification of other pertinent literature. The literature search was conducted from 2 February 2015 to the 6 April 2015. Two independent researchers were involved in the selection of studies and extraction of data. Disagreements between the researchers were resolved through discussion.

View this table:
[Table 1](http://bmjopen.bmj.com/content/6/4/e010054/T1)

Table 1 
Key word search

A summary of inclusion and exclusion criteria is presented in table 2. The timescale of searches was limited from 1999 onwards (ie, the past 15 years) in the interest of identifying the most up-to-date and relevant information. Outcome variables were not predefined in this study, considering that it is exploratory rather than hypothesis-led research. A structured appraisal tool and data extraction form were designed for and used in this study, based on guidelines provided by the UK National Health Service (NHS) Centre for Reviews and Dissemination23 (see online supplementary appendices 1 and 2). Textual narrative synthesis was used for data analysis. This approach enabled analysis of the relationships within and between studies, as well as assessing gaps in the literature.24 ,25

### Supplementary appendix 1

[[bmjinnov-2015-000112supp_appendix1.pdf]](pending:yes)

### Supplementary appendix 2

[[bmjinnov-2015-000112supp_appendix2.pdf]](pending:yes)

View this table:
[Table 2](http://bmjopen.bmj.com/content/6/4/e010054/T2)

Table 2 
Summary of inclusion and exclusion criteria

## Findings

One hundred and sixty-eight citations were identified from searching the electronic databases. All of the identified literature was entered into Endnote and 59 duplicates removed. A thorough review of titles and abstracts was conducted for the remaining 109 studies, of which 96 did not meet the inclusion criteria. Full-text documents were retrieved for 15 articles, including two articles that were discerned by reviewing the reference lists of relevant literature. Of these articles, eight were excluded for failure to meet the inclusion criteria. Seven studies were included in the final review. See figure 1 for an overview of the search strategy. Baseline data of the included studies are summarised in table 3.

View this table:
[Table 3](http://bmjopen.bmj.com/content/6/4/e010054/T3)

Table 3 
Baseline data of the included studies7–13

![Figure 1](http://bmjopen.bmj.com/https://bmjopen.bmj.com/content/bmjopen/6/4/e010054/F1.medium.gif)

[Figure 1](http://bmjopen.bmj.com/content/6/4/e010054/F1)

Figure 1 
Overview of search strategy.

### Assessment of study quality

#### Weaknesses and biases

Of the studies included in this review, a number of weaknesses and biases have been identified. Five studies recruited participants by non-random selection.7–9 ,12 ,13 This method of selection is associated with an increased risk of sampling bias. A recurring problem was not collecting data from all UK medical schools. This is important as curriculum variation means that students' experiences may be different from one school to the next. Moreover, with the exception of Lee *et al*,12 details about the medical schools involved in studies were not provided. All of the studies included in this review relied on questionnaires to collect data7–13 which limited the opportunity for participants to express their perspectives. The response rate varied greatly between studies and in general tended to be higher for those with the least number of participants. No explanation was given for why some participants chose not to participate. In some studies,7 ,9–11 the authors did not provide sufficient information in relation to their methodology such as the type of questions that participants were asked7 ,9 or the method of random selection10 ,11 and data analysis.10

#### Reliability and validity

In three studies, the questionnaire involved the use of a Likert scale.8 ,12 ,13 The items that made up these scales were inter-related, contributing to the internal reliability of the findings. In the study conducted by Khan and Saeed,7 the questionnaires provided a free-text option, with responses obtained from students straight reported in an online supplementary appendix. This enabled independent evaluation and avoided any subjective judgement on behalf of the authors, contributing to the reliability of the study. However, in two studies, participants were asked to provide estimates;10 ,11 this type of questioning is associated with a high degree of subjectivity and inconsistency, reducing the reliability of the findings. Chawdhary *et al*9 provided insufficient details to enable proper assessment of reliability. For example, it is not clear what type of questionnaire was used or the questions that participants were asked.

A recurrent problem was not obtaining data from all UK medical schools. This reduces the external validity of the findings. Regarding internal validity, most of the studies presented sufficient data to support their findings which involved the use of figures and tables.7 ,8 ,10–13 Five of the studies included in this review provided a copy of the questions that participants were asked;8 ,10–13 these appear to have good face validity based on their clarity and appropriateness.

### Themes

From the studies included in this review, four main themes were identified with regard to the provision of undergraduate education for preparedness in ENT: confidence in managing patients, duration of clinical rotations, teaching delivery and student assessment.

#### Confidence in managing patients

Three studies have reported that the majority of final year medical students and clinicians were not confident to assess and manage ENT patients.8–10 Chawdhary *et al*9 conducted a study involving 311 final year medical students, recruited from a UK medical school after the completion of a week-long rotation in ENT. The proportion who did not feel adequately prepared to deal with common ENT symptoms and emergencies was 72% and 77%, respectively. In a study conducted by Clamp *et al*10 involving 357 general practitioners (GPs) from south-west England, over three-quarters of participants felt their ENT undergraduate training had not adequately prepared them for clinical practice. Powell *et al*8 compared the confidence levels of 444 Foundation Year 1 doctors in ENT and cardiovascular medicine. A 10-point Likert scale was used to score the confidence level of participants who represented all 30 UK medical schools. This study found the median confidence of Foundation Year 1 doctors in history taking and performing clinical examinations to be significantly lower for ENT when compared with cardiovascular medicine (p<0.001). Similarly, the median confidence of Foundation Year One doctors in patient management was significantly reduced for ENT when compared with cardiovascular medicine (p<0.001). Furthermore, in the study conducted by Khan and Saeed,7 the opinions of 163 ENT consultants were sought. Over three-quarters of the participants felt that junior doctors were not proficient in the management of patients with common ENT problems.

#### Duration of clinical rotations

Participants in three of the included studies expressed a need for more ENT training in the undergraduate medical programme.7 ,8 ,11 Khan and Saeed7 included 518 students, recruited from nine UK medical schools. Forty-seven per cent of these participants were not offered a formal ENT rotation. Of those who were, the average duration was 8 days. Over one-third of medical students who had undergone a formal ENT rotation felt that their expectations had not been met. Khan and Saeed7 also explored the views of consultant ENT surgeons who felt that the low level of proficiency among junior doctors was due to the limited time assigned to ENT undergraduate training and that longer rotations were needed.

Sharma *et al*11 conducted a study involving 20 senior house officers working in emergency departments across the UK, representing 16 of the then 27 UK medical schools. Only eight doctors (40%) had a formal ENT rotation, with two doctors having received no ENT undergraduate training at all. The mean duration of ENT undergraduate training was 13.7 days. Three-quarters of participants felt that they had not received enough ENT training as an undergraduate. In the study conducted by Powell *et al*,8 involving 444 Foundation Year One doctors from 30 different medical schools, the mean duration of ENT undergraduate training was 3 days of preclinical teaching and 5 days of department exposure. Sixteen per cent of respondents received no experience within an ENT department. Sixty-six per cent of respondents felt that they had not received enough ENT undergraduate training.

#### Teaching delivery

Two studies included in this review found that teaching modalities perceived by students to have the greatest educational value were not always the most commonly used ones in ENT undergraduate training.8 ,9 Chawdhary *et al*9 conducted a study involving 311 final year medical students who had undergone a week-long rotation in ENT. During this period, students spent most of their time in clinics, followed by self-study and then operating theatres. In terms of educational value, students ranked clinics first, tutorials second and operating theatres fifth out of six possible options. Information about the other teaching options is not available. In the study conducted by Powell *et al*,8 the teaching modalities that were most commonly used during ENT undergraduate training were clinics, followed by theatre sessions and then tutorials. Participants ranked formal teaching sessions involving real ENT patients first for educational value—a teaching method that was reported to be the ninth most commonly used, out of a total of 13 options. ENT clinics were ranked second and tutorials third; theatre sessions were ranked seventh for educational value.

Powell *et al*8 also asked participants to score the different types of personnel involved in delivering the ENT undergraduate programme, with the consultants and middle grades scoring highest.

Although students did not rank theatre sessions first for educational value, Lee *et al*12 found that the majority of medical students perceived it to be a beneficial and important part of the undergraduate programme. In this questionnaire-based study, 152 fourth year medical students were recruited from Dundee University after completing a 2-week ENT rotation. On a seven-point Likert scale, student satisfaction of educational needs in operating theatre teaching was rated 3.9 (95% CI 3.7 to 4.2). The importance of theatre attendance as part of the ENT undergraduate rotation was rated 4.7 (95% CI 3.7 to 4.2).

In a non-randomised controlled study, Hajioff and Birchall13 explored factors influencing the level of student satisfaction in ENT clinics. This study involved 25 students, recruited from a UK medical school during their 2-week ENT rotation. Ninety-two per cent of students said that they liked to spend time alone with patients. Student satisfaction with clinics was reported to be significantly higher when they got to see patients alone (74.6%±2.3% (mean±SE) vs 64.2%±2.2%; p=0.0021).

#### Student assessment

Two studies reported medical students having completed their ENT undergraduate training without any formal assessment.8 ,10 In the study conducted by Powell *et al*,8 30% of respondents reported no formal assessment of their ENT knowledge and skills. Similarly, in the study conducted by Clamp *et al*,10 only 21% of ENT undergraduate programmes were found to incorporate a formal examination. However, Clamp *et al*10 also found that the proportion of participants who were satisfied with their undergraduate training rose from 24% to 33% in a subgroup who had undergone a formal examination in ENT.

## Discussion

This work represents the first attempt to review systematically the available evidence pertaining to the provision of undergraduate education for preparedness in ENT, as perceived by medical students and clinicians in the UK. One important aspect highlighted by this review is the paucity of evidence available in this area of interest; this limited the authors' rigour with regard to the quality of the selected studies. Nonetheless, identifying four common themes allowed the authors to collate the results in a structured manner and draw conclusions that are meant to represent a starting point for further and more robust research.

A consistent finding in this review was that the majority of final year medical students and junior doctors did not feel confident to assess and manage ENT patients.7–13 Consultant ENT surgeons, working alongside junior doctors, also felt they were not adequately skilled to deal with common ENT problems.7 There is a general perception that postgraduate trainees in disciplines such as general practice will receive formal training in ENT. However, Clamp *et al*10 found that 30% of GPs in south-west England had received no hospital experience or any form of postgraduate teaching in ENT. This problem of preparedness for ENT is not specific to the UK. A questionnaire-based study conducted by Lennon *et al*26 involving 473 GPs working in Ireland found that three-quarters of the participants felt that they had not received adequate ENT training at undergraduate level. Other studies involving primary healthcare practitioners in America and Canada have found that the majority of doctors were not aware of appropriate referral indications for common ENT procedures such as tonsillectomy.27–29

Medical students and clinicians have expressed the need for longer ENT rotations at undergraduate level. This is also supported by the study conducted by Lennon *et al*26 in Ireland, which demonstrated a statistically significant relationship between the length of training and the level of satisfaction of training received (p<0.001); the level of satisfaction increased as the length of ENT undergraduate training received increased.

Teaching sessions involving ENT patients and delivered by consultant ENT surgeons were perceived by students to have the greatest educational value.8 ,9 These findings are supported by a recent qualitative study conducted by Illing *et al*17 involving 60 newly qualified doctors from three UK medical schools (20 from each school) with a different curricula. The opportunity for hands-on learning in a clinical environment and having a role that enabled supervised engagement with patients were identified as important factors influencing preparedness for practice. Furthermore, although theatre sessions did not rank first in educational value, students perceived this teaching method to be a beneficial and important part of the undergraduate curriculum.12 Ravindra *et al*30 in 2013 conducted a study involving 209 graduates from a UK medical school and found that providing students with the opportunity for active participation while scrubbed, irrespective of how important the task is, significantly increases their level of satisfaction. Prioritising the use of teaching methods and providers that are perceived by students to be of greatest education value is likely to result in a higher level of satisfaction and increased preparedness for ENT.

The level of satisfaction for ENT undergraduate training was higher in a subgroup of doctors who had undergone a formal assessment in ENT.10 However, not all UK medical schools offer students a formal ENT assessment.8 ,10 This is supported by Evans *et al*31 who conducted a study involving 26 newly qualified doctors and found that the level of confidence in performing clinical skills significantly improved for those that were practised in a clinical setting and formally assessed.

### Recommendations for increased preparedness in ENT

From the findings of this review, it is evident that medical students want a formal ENT rotation of more than 2 weeks duration. Nonetheless, the authors realise that it is difficult to allocate an equitable proportion of the undergraduate curriculum to a relatively ‘small’ specialty, particularly when competing against larger disciplines such as General Surgery and Internal Medicine. To help overcome the time constraints placed on the undergraduate curriculum, an increasing number of medical schools are adopting an integrated approach towards teaching and learning medical and surgical specialties.32 This includes the use of simulation training and online virtual learning, alongside clinical rotations.32 Employing such an approach towards ENT would not only be time effective but would provide the opportunity for hands-on learning and formal assessment, factors that are considered important in preparing medical students for clinical practice.

## Conclusions

This systematic literature review provides a starting point to better understand the provision of undergraduate education for preparedness in ENT. The majority of final year medical students and junior doctors are not confident in managing patients with common ENT problems. The findings of this review suggest the need for further development of the ENT undergraduate curricula across the UK. However, there is insufficient evidence from which to draw strong conclusions; this in itself is beneficial as it highlights a gap in the existing literature and supports the need for primary research.

## Footnotes

*   Contributors GRF and IAB made a substantial contribution to the study design, data collection and interpretation of the literature; they produced the first draft of the article. MS made a substantial contribution to the original concept and design of the study, and revised of the first draft of the paper. All authors approved the final manuscript for submission.

*   Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

*   Competing interests None declared.

*   Provenance and peer review Not commissioned; externally peer reviewed.

*   Data sharing statement No additional data are available.

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/)

## References

1.  Powell S. So you want to be an ENT surgeon. [http://careers.bmj.com/careers/advice/view-article.html?id=2391](http://careers.bmj.com/careers/advice/view-article.html?id=2391) (accessed 6 Apr 2015).
    
    

2.  Griffiths E. Incidence of ENT problems in general practice. J R Soc Med 1979;72:740–2.
    
    [FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6MzoiUERGIjtzOjExOiJqb3VybmFsQ29kZSI7czo1OiJzcGpycyI7czo1OiJyZXNpZCI7czo5OiI3Mi8xMC83NDAiO3M6NDoiYXRvbSI7czoyNToiL2Jtam9wZW4vNi80L2UwMTAwNTQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

3.  Morris PD, Pracy R. Training for ENT problems in general practice. Practitioner 1983;227:995–9.
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=6889265&link_type=MED&atom=%2Fbmjopen%2F6%2F4%2Fe010054.atom) 

4.  Donnelly MJ, Quraishi MS, McShane DP. ENT and general practice: a study of paediatric ENT problems seen in general practice and recommendations for general practitioner training in ENT in Ireland. Ir J Med Sci 1995;164:209–11. [doi:10.1007/BF02967831](http://dx.doi.org/10.1007/BF02967831)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1007/BF02967831&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=7672936&link_type=MED&atom=%2Fbmjopen%2F6%2F4%2Fe010054.atom) 

5.  General Medical Council. Tomorrow's doctors. London: General Medical Council, 2009.
    
    

6.  General Medical Council. General Medical Council Education Strategy 2011–2013. London: General Medical Council, 2013.
    
    

7.  Khan M, Saeed SR. Provision of undergraduate otorhinolaryngology teaching within general medical council approved UK medical schools: What is current practice? Clin Otolaryngol 2012;126:340–4. [doi:10.1017/S0022215111003379](http://dx.doi.org/10.1017/S0022215111003379)
    
    

8.  Powell J, Cooles FA, Carrie S, et al. Is undergraduate medical education working for ENT surgery? A survey of UK medical school graduates. J Laryngol Otol 2011;125:896–905. [doi:10.1017/S0022215111001575](http://dx.doi.org/10.1017/S0022215111001575)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1017/S0022215111001575&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=21774848&link_type=MED&atom=%2Fbmjopen%2F6%2F4%2Fe010054.atom) 

9.  Chawdhary G, Ho EC, Minhas SS. Undergraduate ENT education: what students want. Clin Otolaryngol 2009;34:584–5. [doi:10.1111/j.1749-4486.2009.02027.x](http://dx.doi.org/10.1111/j.1749-4486.2009.02027.x)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1111/j.1749-4486.2009.02027.x&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=20070780&link_type=MED&atom=%2Fbmjopen%2F6%2F4%2Fe010054.atom) 

10. Clamp PJ, Gunasekaran S, Pothier DD, et al. ENT in general practice: training, experience and referral rates. J Laryngol Otol 2007;121:580–3. [doi:10.1017/S0022215106003495](http://dx.doi.org/10.1017/S0022215106003495)
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=17052364&link_type=MED&atom=%2Fbmjopen%2F6%2F4%2Fe010054.atom) 

11. Sharma A, Machen K, Clarke B, et al. Is undergraduate otorhinolaryngology teaching relevant to junior doctors working in accident and emergency departments? J Laryngol Otol 2006;120:949–51. [doi:10.1017/S0022215106002246](http://dx.doi.org/10.1017/S0022215106002246)
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=16879761&link_type=MED&atom=%2Fbmjopen%2F6%2F4%2Fe010054.atom) 

12. Lee MSW, Montague ML, Hussain SSM. Student-perceived benefit from otolaryngology theatre attendance. J Laryngol Otol 2005;119:293–8. [doi:10.1258/0022215054020412](http://dx.doi.org/10.1258/0022215054020412)
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=15949084&link_type=MED&atom=%2Fbmjopen%2F6%2F4%2Fe010054.atom) 

13. Hajioff D, Birchall M. Medical students in ENT outpatient clinics: appointment times, patient satisfaction and student satisfaction. Med Educ 1999;33:669–73. [doi:10.1046/j.1365-2923.1999.00422.x](http://dx.doi.org/10.1046/j.1365-2923.1999.00422.x)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1046/j.1365-2923.1999.00422.x&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=10476017&link_type=MED&atom=%2Fbmjopen%2F6%2F4%2Fe010054.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000082712100007&link_type=ISI) 

14. Goldacre MJ, Lambert T, Evans J, et al. PRHOs’ views on whether their medical school had trained them well. BMJ 2003;326:1011–101. [doi:10.1136/bmj.326.7397.1011](http://dx.doi.org/10.1136/bmj.326.7397.1011)
    
    [FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEzOiIzMjYvNzM5Ny8xMDExIjtzOjQ6ImF0b20iO3M6MjU6Ii9ibWpvcGVuLzYvNC9lMDEwMDU0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

15. Cave J, Goldacre M, Lambert T, et al. Newly qualified doctors’ views about whether their medical school had trained them well. BMC Med Educ 2007;7:50. [doi:10.1186/1472-6920-7-50](http://dx.doi.org/10.1186/1472-6920-7-50)
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=18053231&link_type=MED&atom=%2Fbmjopen%2F6%2F4%2Fe010054.atom) 

16. Wall D, Bolshaw A, Carolan J. From undergraduate medical education to pre-registration house officer: how prepared are students? Med Teach 2006;28:435–9. [doi:10.1080/01421590600625171](http://dx.doi.org/10.1080/01421590600625171)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1080/01421590600625171&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=16973456&link_type=MED&atom=%2Fbmjopen%2F6%2F4%2Fe010054.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000241164600007&link_type=ISI) 

17. Illing JC, Morrow GM, Rothwell nee Kergon CR, et al. Perceptions of UK medical graduates’ preparedness for practice: a multi-centre qualitative study reflecting the importance of learning on the job. BMC Med Educ 2013;13:34. [doi:10.1186/1472-6920-13-34](http://dx.doi.org/10.1186/1472-6920-13-34)
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=23446055&link_type=MED&atom=%2Fbmjopen%2F6%2F4%2Fe010054.atom) 

18. Matheson C, Matheson D. How well prepared are medical students for their first year as doctors? The views of consultants and specialist registrars in two teaching hospitals. Postgrad Med J 2009;85:582–9. [doi:10.1136/pgmj.2008.071639](http://dx.doi.org/10.1136/pgmj.2008.071639)
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTI6InBvc3RncmFkbWVkaiI7czo1OiJyZXNpZCI7czoxMToiODUvMTAwOS81ODIiO3M6NDoiYXRvbSI7czoyNToiL2Jtam9wZW4vNi80L2UwMTAwNTQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

19. Tallentire VR, Smith SE, Skinner J, et al. The preparedness of UK graduates in acute care: a systematic literature review. Postgrad Med J 2012;88:365–71. [doi:10.1136/postgradmedj-2011-130232](http://dx.doi.org/10.1136/postgradmedj-2011-130232)
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTI6InBvc3RncmFkbWVkaiI7czo1OiJyZXNpZCI7czoxMToiODgvMTA0MS8zNjUiO3M6NDoiYXRvbSI7czoyNToiL2Jtam9wZW4vNi80L2UwMTAwNTQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

20. Langdale LA, Schaad D, Wipf J, et al. Preparing graduates for the first year of residency: are medical schools meeting the need? Acad Med 2003;78:39–44. [doi:10.1097/00001888-200301000-00009](http://dx.doi.org/10.1097/00001888-200301000-00009)
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=12525408&link_type=MED&atom=%2Fbmjopen%2F6%2F4%2Fe010054.atom) 

21. Hilmer SN, Seale JP, Le Couteur DG, et al. Do medical courses adequately prepare interns for safe and effective prescribing in New South Wales public hospitals? J Intern Med 2009;39:428–34. [doi:10.1111/j.1445-5994.2009.01942.x](http://dx.doi.org/10.1111/j.1445-5994.2009.01942.x)
    
    

22. Armstrong EG, Mackey M, Spear SJ. Medical education as a process management problem. Acad Med 2004;79:721–8. [doi:10.1097/00001888-200408000-00002](http://dx.doi.org/10.1097/00001888-200408000-00002)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1097/00001888-200408000-00002&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=15277126&link_type=MED&atom=%2Fbmjopen%2F6%2F4%2Fe010054.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000223002000002&link_type=ISI) 

23. Centre for Reviews and Dissemination. Systematic reviews: CRD's guidance for undertaking reviews in health care. University of York, 2009.
    
    

24. Centre for Reviews and Dissemination. Guidance on the conduct of narrative synthesis in systematic reviews. University of York, 2009.
    
    

25. Lucas PJ, Baird J, Arai L, et al. Worked examples of alternative methods for the synthesis of qualitative and quantitative research in systematic reviews. BMC Med Res Methodol 2007;7:4. [doi:10.1186/1471-2288-7-4](http://dx.doi.org/10.1186/1471-2288-7-4)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1186/1471-2288-7-4&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=17224044&link_type=MED&atom=%2Fbmjopen%2F6%2F4%2Fe010054.atom) 

26. Lennon P, O'Donovan JP, O'Donoghue S, et al. The otolaryngology, head and neck training appraisal questionnaire: a national general practice perspective. Ir J Med Sci 2013;182:609–14. [doi:10.1007/s11845-013-0937-7](http://dx.doi.org/10.1007/s11845-013-0937-7)
    
    

27. Hu A, Sardesai MG, Meyer TK. A need for otolaryngology education among primary care providers. Med Educ Online 2012;17:17350. [doi:10.3402/meo.v17i0.17350](http://dx.doi.org/10.3402/meo.v17i0.17350)
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=22754276&link_type=MED&atom=%2Fbmjopen%2F6%2F4%2Fe010054.atom) 

28. Carr MM, Brown DH, Reznick RK. Needs assessment for an undergraduate otolaryngology curriculum. Otolaryngol Head Neck Surg 1999;120:865–8. [doi:10.1016/S0194-5998(99)70328-1](http://dx.doi.org/10.1016/S0194-5998(99)70328-1)
    
    

29. Carr MM, Kolenda J, Clarke KD. Tonsillectomy: indications for referral by family physicians versus indications for survey by otolaryngologists. J Otolaryngol 1997;26:225–8.
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=9263889&link_type=MED&atom=%2Fbmjopen%2F6%2F4%2Fe010054.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=A1997XQ50800002&link_type=ISI) 

30. Ravindra P, Fitzgerald JE, Bhangu A, et al. Quantifying factors influencing operating theater teaching, participation, and learning opportunities for medical students in surgery. J Surg Educ 2013;70:495–501. [doi:10.1016/j.jsurg.2013.02.011](http://dx.doi.org/10.1016/j.jsurg.2013.02.011)
    
    

31. Evans DE, Wood DF, Roberts CM. The effect of an extended hospital induction on perceived confidence and assessed clinical skills of newly qualified pre-registration house officers. Med Educ 2004;38:998–1001. [doi:10.1111/j.1365-2929.2004.01908.x](http://dx.doi.org/10.1111/j.1365-2929.2004.01908.x)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1111/j.1365-2929.2004.01908.x&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=15327682&link_type=MED&atom=%2Fbmjopen%2F6%2F4%2Fe010054.atom) 

32. Howe A, Campion P, Searle J, et al. New perspectives—approaches to medical education at four new UK medical schools. BMJ 2004;329:327–32. [doi:10.1136/bmj.329.7461.327](http://dx.doi.org/10.1136/bmj.329.7461.327)
    
    [FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMjkvNzQ2MS8zMjciO3M6NDoiYXRvbSI7czoyNToiL2Jtam9wZW4vNi80L2UwMTAwNTQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9)