Doctors are good at standardized tests. It remains unclear whether those tests are good for the doctors. Since 1980, the American Board of Emergency Medicine (ABEM) has set forth the emergency medicine board certification processes, including the requirement of an oral examination in addition to a written qualifying exam.¹ In response to the coronavirus pandemic, ABEM moved the oral boards component online to a Zoom format.² This restructuring provides an opportunity to question the value and validity of the test itself. ABEM’s oral exam is expensive and has never been shown to predict an emergency physician’s clinical competence; its 2020 reformatting should be the first step in abandoning the exam altogether.

Cost

Money talks. Or rather, in this case, talking generates money for ABEM. As of November 2021, the registration fee for the ABEM oral boards is $1,255 if paid early and $1,565 if paid late. This fee can only be paid after the examinee has paid the initial certification fee ($420 early, $840 late) and has paid for and passed the written qualifying exam ($960 early, $1,260 late).³ Nearly 2,500 board-eligible emergency medicine residents graduate every year.^4,5 Thousands of doctors paying thousands of dollars in fees adds up. Indeed, fully 34 percent of ABEM’s net $2.1 million revenue in FY19 came from these initial certification costs, and these sums include none of the continuing certification fees that ABEM extracts from practitioners on an ongoing basis once board-certified.⁶

ABEM will argue that the high fees are necessary to cover the overhead costs of administering the oral exam, and this may be true. Even after converting to a Zoom format and presumably saving on hotel costs (the organization having previously faced criticism for lavish lodgings), ABEM still has to produce the test material; administer the exam; analyze testing results; and support a host of administrative, clerical, and information technology staff.⁷ The net profit to ABEM after accounting for these expenditures is probably small, if anything at all. But the cost to physicians is real, as is the unpaid time required to study for and take the test. The question is whether the benefits of the exam justify these costs.

Benefit

Ideally, board certification should distinguish exemplary practitioners of emergency medicine. Indeed, ABEM’s board of directors notes, “The purpose of initial certification is to objectively and independently confirm that physicians who complete an emergency medicine residency demonstrate core knowledge, skills, and abilities needed to practice emergency medicine at the highest standards.”⁸ The problem is that ABEM has no evidence to show that it can make this determination.

Despite 40 years of oral boards testing, there simply are no data supporting the oral exam as an accurate means for differentiating who is a safe, competent emergency doctor and who is not. There are some data weakly supportive of board certification in general. Hospitals with more board-certified emergency physicians may miss fewer myocardial infarctions, and lapses in continuous board certification are associated with increased risk of state medical board disciplinary action.^9,10 Both of these studies are fraught with confounders, and neither teases apart which component of board certification—whether written or oral or recertification—is important. In fact, the scant data that do exist suggest one’s board exam scores (either written or oral) do not correlate significantly with one’s clinical effectiveness as measured by patients per hour.¹¹ At best, one can construct a syllogism to suggest that written standardized examinations may predict later clinical performance. USMLE Step 2 CK exam scores have been shown to predict one’s odds of passing the ABEM written qualifying exam.¹² And studies have shown that Step 2 CK scores predict international medical graduates’ patient mortality from acute myocardial infarction and congestive heart failure and predict U.S. medical graduates’ odds of receiving disciplinary action from a state medical board.^13,14 Possibly then, if the syllogism holds, doing well on the ABEM written exam could translate into later clinical success, though no study has ever directly measured this. No such stretches of logic can be made regarding the oral exam.

Perhaps recognizing this nonexistent predictive value of the oral exam, ABEM boasts research about the exam’s reliability. However, of the six studies ABEM cites for this purpose, only two date from the current millennium.^15–20 These studies do suggest that the oral boards exam is an internally valid instrument, though it remains unclear how well an exam based on person-to-person interactions holds up in an era of increased attention to implicit biases. Even if the exam is technically consistent and completely devoid of unconscious bias, creating a reliable exam is not the same as creating a useful one. Producing 10,000 cars that consistently pull to the left when steered down a straight road demonstrates that one’s production process is reliable, but the final product is still a clunker.

Analysis

In light of these costs and dubious benefit, one might reasonably question ABEM’s continued insistence on the oral exam. Of the 40 specialties recognized by the American Board of Medical Specialties, only 19 require an oral examination, and 11 of those 19 are surgical specialties.²¹ Why does emergency medicine, a nonsurgical specialty, require an oral component, especially one that is expensive and of unproven utility? Is there a better way forward?

ABEM justifies the oral exam because the written exam is incomplete. The organization explains, “The oral exam measures different competencies and dimensions than the qualifying (written) exam. At least 36 percent of the knowledge, skills, and abilities that ABEM assesses are only measured by the oral exam.”²² To continue the car analogy, this is the equivalent of selling 64 percent of a vehicle and then requiring customers to return to the dealer at a later date to purchase the other 36 percent. The answer to an incomplete written exam is not an additional oral exam but rather a better written one.

The logical strategy would be to abandon the oral examination completely and rely solely on the written examination to determine board certification. As noted earlier, one can string together some evidentiary basis for written exams predicting clinical performance. And simple statistical models based on a few objective data points have been shown to score better than individual interviewers at predicting the success of applicants in a range of endeavors from school performance to military roles to recidivism among juvenile offenders.²³ Simply put, we may not need the oral exam to predict who will become a good and safe emergency physician.

If ABEM insists on keeping the oral exam as an element of board certification, then the test should remain online. In the wake of COVID-19 lockdowns, industries worldwide have moved to remote working. Why should our testing be any different? Surely, we can respond to an examiner’s clinical cases as easily through a webcam as across a desk in a hotel room. This tele-testing approach would reduce the overhead costs of administering the exam, and those cost savings could then be passed on to examinees, who are usually fresh graduates laden with student loan debt. Or perhaps ABEM could use the money saved to conduct research trials to show that its oral examination actually accomplishes what it sets out to do.

Conclusion

For 40 years, ABEM has forced emergency physicians to pay dearly for an inconvenient oral examination that is unsupported by evidence. COVID-19 provides the perfect opportunity for our specialty to leave this exam component in the past. At the very least, cheaper virtual boards should be the path forward. As emergency medicine doctors, we do not perform interventions on our patients without robust evidence to justify the clinical action, and it is time we hold our board certification process to the same standard.

Dr. Schwartz is a board-certified emergency physician with a background in public health policy whose previous research interests include helicopter air ambulance billing, LGBT rights, firearm violence, and sex education in schools. He is currently on sabbatical.

Dr. Babineau is a board-certified emergency physician and assistant professor of emergency medicine at the Dartmouth-Hitchcock Medical Center in Lebanon, New Hampshire. His academic interests include resident education and clinical decision making.

References

1. ABEM history. American Board of Emergency Medicine website. Accessed Nov. 19, 2021.
2. ABEM virtual oral exam starting in December 2020. American Board of Emergency Medicine website. Accessed Nov. 19, 2021.
3. Exams. American Board of Emergency Medicine website. Accessed Nov. 19, 2021.
4. Blazar E. Emergency medicine doesn’t need more residencies. Emerg Med News. 2019;41(2):6-7.
5. Haas MRC, Hopson LR, Zink BJ. Too big too fast? Potential implications of the rapid increase in emergency medicine residency positions. AEM Edu Train. 2019;4(S1):S13-S21.
6. ABEM Annual Report 2019–2020. American Board of Emergency Medicine website. Accessed Nov. 19, 2021.
7. Fiore K, Henderson J, Basen R. Continuing certification a cash cow for emergency medicine board. Medpage Today website. Accessed Nov. 19, 2021.
8. Become certified. American Board of Emergency Medicine website. Accessed Nov. 19, 2021.
9. Wilson M, Welch J, Schuur J, et al. Hospital and emergency department factors associated with variations in missed diagnosis and costs for patients age 65 years and older with acute myocardial infarction who present to emergency departments. Acad Emerg Med. 2014;21(10):1101-1108.
10. Nelson LS, Duhigg LM, Arnold GK, et al. The association between maintaining American Board of Emergency Medicine certification and state medical board disciplinary actions. J Emerg Med. 2019;57(6):772-779.
11. Frederick RC, Hafner JW, Schaefer TJ, et al. Outcome measures for emergency medicine residency graduates: do measures of academic and clinical performance during residency training correlate with American Board of Emergency Medicine test performance? Acad Emerg Med. 2011;18(S2):S59-S64.
12. Caffery T, Fredette J, Musso MW, et al. Predicting American Board of Emergency Medicine qualifying examination passage using United States Medical Licensing Examination Step Scores. Ochsner J. 2018;18(3):204-208.
13. Norcini JJ, Boulet JR, Opalek A, et al. The relationship between licensing examination performance and the outcomes of care by international medical school graduates. Acad Med. 2014;89(8):1157-1162.
14. Cuddy MM, Young A, Gelman A, et al. Exploring the relationships between USMLE performance and disciplinary action in practice: a validity study of score inferences from a licensure examination. Acad Med. 2017;92(12):1780-1785.
15. Bianchi L, Gallagher EJ, Korte R, et al. Interexaminer agreement on the American Board of Emergency Medicine oral certification examination. Ann Emerg Med. 2003;41(6):859-864.
16. Kowalenko T, Heller BN, Strauss RW, et al. Initial validity analysis of the American Board of Emergency Medicine enhanced oral examination. Acad Emerg Med. 2017;24(1):125-129.
17. Maatsch JL, Munger BS, Podgorny G. On the reliability and validity of the board examination in emergency medicine. In: Wolcott BA, Rund DA, eds. Emergency Medicine Annual: 1982. Norwalk, Conn.: Appleton-Century Crofts; 1982: 183-222.
18. Munger BS, Krome RL, Maatsch JC, et al. The certification examination in emergency medicine: an update. Ann Emerg Med. 1982;11(2):91-96.
19. Reinhart MA. Advantages to using the oral examination. In: Mancall EL, Bashook PG, eds. Assessing clinical reasoning: the oral examination and alternative methods. Evanston, IL: American Board of Medical Specialties; 1995:31-39.
20. Solomon DJ, Reinhart MA, Bridgham RG, et al. An assessment of an oral examination format for evaluating clinical competence in emergency medicine. Acad Med. 1990;65(9 Suppl):S43-44.
21. ABMS board certification report 2019-2020. American Board of Medical Specialties website. Accessed Nov. 19, 2021.
22. ABEM virtual oral exam: frequently asked questions. American Board of Emergency Medicine website. Accessed Nov. 19, 2021.
23. Kahneman D. Thinking, Fast and Slow. New York: Farrar, Straus, and Giroux; 2011.