USMLE Step 1 Scores as a Significant Predictor of Future Board Passage in Pediatrics

doi:10.1016/j.ambp.2007.01.002

Ambulatory Pediatrics

Volume 7, Issue 2, 1 March 2007, Pages 192-195

https://doi.org/10.1016/j.ambp.2007.01.002 Get rights and content

Purpose

To explore the relationship(s) between USMLE, In-Training Exam, and American Board of Pediatrics (ABP) board-certifying exam scores within a Pediatric residency-training program.

Methods

Data were abstracted from records of graduating residents from the Pediatric residency program at the University of Florida College of Medicine Jacksonville from 1999 to 2005. Seventy (70) residents were identified and their files reviewed for the following information: USMLE Step 1 and 2 scores, in-training exam results and eventual board scores as reported by the ABP. Correlation and regression analyses were performed and compared across all tests.

Results

The correlation coefficients between the three types of tests were all statistically significant. Using logistic regression, however, only USMLE Step 1 scores (compared to Step 2) had a statistically significant association with board performance. Interestingly, none of the three in-training exam scores had any additional impact on predicting board performance given one’s USMLE Step 1 score. USMLE Step 1 scores greater than 220 were associated with nearly a 95% passage rate on the board-certifying exam.

Conclusions

The data suggests that performance on USMLE Step 1 is an important predictor of a resident’s chances of passing the pediatric boards. This information, which is available when a resident initiates training, can be used to identify those at risk of not passing the boards. While Step 1 scores should not be used as a sole determinant in the recruiting process, individual learning plans can be developed and implemented early in training to maximize one’s ability to pass the certifying exam.

Section snippets

Subjects

The University of Florida College of Medicine sponsors a pediatric residency program at its urban campus located in Jacksonville, Florida, which enrolls eleven residents at each training level. Data for this study were abstracted from the records of graduating residents of the program dating from to 1999-2005. No funding was necessary to perform this study and a request for an IRB exemption was submitted and approved. In total, seventy (70) residents were identified for possible inclusion in

Results

Between 1999 and 2005, the overall pass rate for first-time takers on the ABP certifying exam was 68%. In order to assess the strength of the relationship between the three types of tests, Pearson correlation coefficients were obtained comparing USMLE Step 1 scores with in-training exam and ABP board scores and all were found to be highly correlated. Similar results were also seen when the same correlations were run using USMLE Step 2 scores (see Table 1). However, when a univariate logistic

Discussion

The performance of graduates on the American Board of Pediatrics certifying exam is an important indication of the residents’ competence and medical knowledge. The Pediatric Residency Review Committee (RRC) considers the performance of graduates on their certifying exam to be an indicator of the quality of the educational program. The average pass rate for residency programs is available to candidates and many use this information to guide their choice of training programs. While previous

Acknowledgment

We would like to acknowledge Dr. Frank Genuardi and Dr. David Wood for their assistance in preparing this manuscript.

References (12)

T.D. Boyse et al.
Does medical school performance predict radiology resident performance?
Acad Radiol
(2002)
T.B. Dodson et al.
Massachusetts General Hospital/Harvard Medical School MD oral and maxillofacial surgery program: a 30-year review
J Oral Maxillofac Surg
(2004)
G.D. Denton et al.
Is a faculty developed pretest equivalent to pre-third year GPA or USMLE step 1 as a predictor of third-year internal medicine clerkship outcomes?
Teach Learn Med
(2004)
J.J. Veloski et al.
Prediction of students’ performances on licensing examinations using age, race, sex, undergraduate GPA’s and MCAT scores
Acad Med
(2000)
M.B. Edmond et al.
Racial bias in using USMLE step 1 scores to grant internal medicine residency interviews
Acad Med
(2001)
J.D. FitzGerald et al.
Didactic teaching conferences for IM residents: who attends, and is attendance related to medical certifying examination scores?
Acad Med
(2003)

There are more references available in the full text version of this article.

Cited by (61)

Predictors of Matching into Anesthesiology and Surgery: Analysis of One Program's Results
2023, Journal of Surgical Education
The objectives of this study were to use a multivariable regression model to determine what application factors made anesthesiology and surgery applicants more or less likely to match into an anesthesiology or surgery residency program.
Surgery and Anesthesiology applicants listed on the final National Resident Matching Program (NRMP) Rank Order Lists from WMC in the 2020-2021 application cycle were included in analysis. All applicant data were collected through the Electronic Residency Application Service (ERAS). All ERAS and letters of recommendation (LOR) data were deidentified and LOR were subsequently inputted into a linguistics software to analyze the language use in LOR. Descriptive analyses were conducted to compare variables between applicants that matched to a specific residency program and those who matched elsewhere. A multivariable regression model was then used to determine characteristics of anesthesiology and surgery applicants that were indicative of matching to a specific rank of residency program.
A total of 116 anesthesiology and 78 surgery applicants were included in final analysis. Analysis of anesthesiology applicants yielded four significant application characteristics that influenced matching to a higher or lower ranked residency program: USMLE Step 2 CK scores, medical school attended, insight category words in LOR, and anger category words in LOR. Similarly, analysis of surgery applicants yielded four significant characteristics: Race, USMLE Step 1 scores, insight category words, and see category words.
Our results demonstrated that specialties of anesthesiology and surgery considered different metrics regarding the residency application process. Among the many factors that were analyzed, USMLE scores and language in LOR were considered significant in both specialties. As the application process continues to evolve, we may see a shift in what application factors are considered more important than others.
The Change of USMLE Step 1 to Pass/Fail: Perspectives of the Surgery Program Director
2021, Journal of Surgical Education
This study sought to evaluate the perspectives of surgical program directors regarding the change of USMLE Step 1 to pass/fail grading.
Validated electronic survey.
Vanderbilt University Medical Center Department of Plastic Surgery.
Program directors of all ACMGE-accredited General Surgery, Integrated Vascular Surgery, Integrated Thoracic Surgery, and Integrated Plastic Surgery residency programs.
The overall response rate was 55.5%. Most PDs (78.1%) disagreed with the scoring change. Only 19.6% believe this change will improve medical student well-being. For 63.5% of PDs, medical school pedigree will become more important, and 52.7% believe it will place international medical graduates at a disadvantage. Only 6.2% believe Step 2 CK should also be pass/fail, while 88.7% will increase the weight of Step 2 CK and 88.4% will now require Step 2 CK score submission with the electronic residency application service.
While well-intentioned, changing USMLE Step 1 to pass/fail may have unintended consequences and may disadvantage certain groups of applicants. The emphasis on Step 1, and resulting test-taking apprehension, will likely shift to Step 2 CK. Proponents of equitable evaluation should direct their efforts toward increasing, not decreasing, the number of objective measures available for student assessment.
The USMLE Step 1 Pass/Fail Reporting Proposal: Another View
2019, Academic Radiology
The Association of Program Directors in Radiology recently issued a statement endorsing continued reporting of results of the United States Medical Licensing Examination (USMLE) as a three-digit score. While this position was approved by the Association of Program Directors in Radiology Board of Directors, it does not reflect the opinions of all radiology program directors. Here, we present an argument in support of reporting USMLE results as pass/fail.
As a psychometric instrument, the USMLE Step 1 is designed to assess basic science knowledge and intended to inform a binary decision on licensure. Due to a steadily-increasing burden of applications to review, program directors have increasingly relied upon scores for candidate screening. Such use has multiple adverse consequences. Student focus on Step 1 systematically devalues educational content not evaluated on the exam, and the reliance on Step 1 scores almost certainly works against efforts to increase workforce diversity. Moreover, the increasing pressure of “Step 1 Mania” has negative consequences for trainee mental health and wellness.
Despite the widespread use of Step 1 scores to select applicants, there are little data to correlate scores to meaningful outcomes related to patient care or clinical practice. We find the current situation untenable, and believe a necessary first step toward reform is making Step 1 a pass/fail only examination.
Predictors of Success on the ABR Core Examination
2019, Journal of the American College of Radiology
The ABR Core Examination tests the comprehensive radiology curriculum. On first attempt, 10% of radiology residents fail. This study investigates factors that predict success or failure on the examination.
An anonymous questionnaire was distributed to all accredited residency programs in diagnostic radiology. Residents who had recently taken the examination answered questions regarding demographics, program, preparation, learning methods, and examination results. Bivariate and multivariate analyses were performed.
The study survey was completed by 273 residents. One-tenth of respondents failed or conditionally passed the examination. Residents were more likely to pass the examination if they had trained at an academic program, scored higher on the US Medical Licensing Examination Step 1 and Step 2, achieved higher in-training examination percentile score, completed more practice questions, practiced with both oral and multiple-choice practice questions, and slept longer per night (P < .05 for all). Success was not associated with utilization of specific study resources or protected study time. Lower total number of months spent studying and lower total number of call-containing weeks before the examination were protective factors on multivariate analysis.
Prior performance on standardized examinations predicts success. Training at an academic institution confers an advantage in performance. The ACR In-Training Examination is an appropriate readiness preparation tool. No specific preparation material or resource can guarantee success. Short-duration, high-quality studying is the most effective method of preparation.
Do Residency Selection Factors Predict Radiology Resident Performance?
2018, Academic Radiology
The purpose of our study is to determine what information in medical student residency applications predicts radiology residency success as defined by objective clinical performance data.
We performed a retrospective cohort study of residents who entered our institution's residency program through the National Resident Matching Program as postgraduate year 2 residents and completed the program over the past 2 years. Medical school grades, selection to Alpha Omega Alpha (AOA) Honor Society, United States Medical Licensing Examination (USMLE) scores, publication in peer-reviewed journals, and whether the applicant was from a peer institution were the variables examined. Clinical performance was determined by calculating each resident's cumulative major discordance rate for on-call cases the resident read and gave a preliminary interpretation. A major discordance was defined as a difference between the preliminary resident and the final attending interpretations that could immediately impact the care of the patient. A multivariate logistic regression was performed to determine significant variables.
Twenty-seven residents provided preliminary reports on call for 67,145 studies. The mean major discordance rate was 1.08% (range 0.34%–2.54%). Higher USMLE Step 1 scores, publication before residency, and election to AOA Honor Society were all statistically significant predictors of lower major discordance rates (P values 0.01, 0.01, and <0.001, respectively).
Overall resident performance was excellent. There are predictors that help select the better performing residents, namely higher USMLE Step 1 scores, one to two publications during medical school, and election to AOA in the junior year of medical school.
United States Medical Licensing Examination and American Board of Pediatrics Certification Examination Results: Does the Residency Program Contribute to Trainee Achievement
2017, Journal of Pediatrics
Citation Excerpt :
In contrast, the contribution of training site to performance is quite modest by comparison. In the pediatric literature, McCaskill et al4 performed a study examining 70 residents' USMLE 1 and ABP results and found that USMLE 1 predicted 45% of the variance in ABP examination results. Swanson et al3 in 2009 examined all 2654 first-time takers of the American Board of Orthopedic Surgery certifying examination between 2002 and 2006.
To determine whether training site or prior examinee performance on the US Medical Licensing Examination (USMLE) step 1 and step 2 might predict pass rates on the American Board of Pediatrics (ABP) certifying examination.
Data from graduates of pediatric residency programs completing the ABP certifying examination between 2009 and 2013 were obtained. For each, results of the initial ABP certifying examination were obtained, as well as results on National Board of Medical Examiners (NBME) step 1 and step 2 examinations. Hierarchical linear modeling was used to nest first-time ABP results within training programs to isolate program contribution to ABP results while controlling for USMLE step 1 and step 2 scores. Stepwise linear regression was then used to determine which of these examinations was a better predictor of ABP results.
A total of 1110 graduates of 15 programs had complete testing results and were subject to analysis. Mean ABP scores for these programs ranged from 186.13 to 214.32. The hierarchical linear model suggested that the interaction of step 1 and 2 scores predicted ABP performance (F[1,1007.70] = 6.44, P = .011). By conducting a multilevel model by training program, both USMLE step examinations predicted first-time ABP results (b = .002, t = 2.54, P = .011). Linear regression analyses indicated that step 2 results were a better predictor of ABP performance than step 1 or a combination of the two USMLE scores.
Performance on the USMLE examinations, especially step 2, predicts performance on the ABP certifying examination. The contribution of training site to ABP performance was statistically significant, though contributed modestly to the effect compared with prior USMLE scores.

View all citing articles on Scopus

View full text

Original articleUSMLE Step 1 Scores as a Significant Predictor of Future Board Passage in Pediatrics

Purpose

Methods

Results

Conclusions

Section snippets

Subjects

Results

Discussion

Acknowledgment

Acad Radiol

J Oral Maxillofac Surg

Is a faculty developed pretest equivalent to pre-third year GPA or USMLE step 1 as a predictor of third-year internal medicine clerkship outcomes?

Teach Learn Med

Prediction of students’ performances on licensing examinations using age, race, sex, undergraduate GPA’s and MCAT scores

Acad Med

Racial bias in using USMLE step 1 scores to grant internal medicine residency interviews

Acad Med

Didactic teaching conferences for IM residents: who attends, and is attendance related to medical certifying examination scores?

Acad Med

Original article
USMLE Step 1 Scores as a Significant Predictor of Future Board Passage in Pediatrics