When the Agency for Healthcare Research and Quality (AHRQ) released its systematic review of diagnostic errors in the emergency department on Dec. 15, 2022, the report immediately started making waves.

The media quickly picked up the 744-page review prepared by the Johns Hopkins University Evidence-based Practice Center and presented it to the public with headlines including “ER Doctors Misdiagnose Patients with Unusual Symptoms” (New York Times, Dec. 15) and “More than 7 million incorrect diagnoses made in US emergency rooms every year, government report finds” (CNN, Dec. 16).

The emergency medicine community also responded swiftly. On Dec. 14, a joint letter from ACEP and nine other emergency medicine organizations stated: “The report makes misleading, incomplete, and erroneous conclusions from the literature reviewed and conveys a tone that inaccurately characterizes and unnecessarily disparages the practice of emergency medicine in the United States.”

As the news circulated, the emergency medicine and research communities dug past the headlines and into the methodology of “Diagnostic Errors in the Emergency Department: A Systematic Review.” Many took to social media to question some of its key findings. Kristen Panthagani, MD, PhD, an emergency medicine resident at Yale University and a self-described “dataviz nerd,” wrote a thorough Twitter thread and a blog post examining the methodology and calling into question the “statistically terrible” extrapolation that resulted in the ensuing headlines.  

Questions soon arose about the report’s peer and public review process. In an article posted to Inside Medicine on Dec. 16 by Jeremy Faust, MD, MS, FACEP, the Johns Hopkins University author team’s private responses to peer reviewer and technical expert comments suggested that several critical comments were not taken into consideration for the final report.  

The Results section of the report is where the critical comments were concentrated, including those questioning the diagnostic error rate extrapolation for the U.S. emergency departments that eventually generated controversy. Technical Expert Panelist 2 wrote: “This section has a fatal flaw and should be removed, specifically, any national extrapolation. Headline grabbing, yes, but this is at best gravely misleading…”

Peer Reviewer 1 stated, “These summarized results should include the limitations of the evidence. There is not good data to make accurate estimates in the US as presented in the summary.” However, while AHRQ instructs authors to address all feedback from peer reviewers, technical reviewers, and the public, the authors are not obligated to incorporate all feedback into the final report, as evidenced by the writing team’s response to reviewers: “We disagree that the data are insufficient to make extrapolations,” they wrote in their response to the reviewer comments. In the body of the final report, the authors stated that “The overall representativeness of this estimate for U.S. ED care is uncertain, but the figure is not outside the range expected.” ACEP Now emailed questions to the study authors about the validity of these extrapolations and received a statement in response that did not address those specific questions.

Mark Graber, an internist and nephrologist who is also Founder and President Emeritus of the Society to Improve Diagnosis in Medicine, has studied diagnostic error for more than 25 years. He served as a peer reviewer for the study. Peer reviewers and technical expert panelist are suggested by the authors and approved by AHRQ.

“It’s an incredibly detailed and extensive study, and it was difficult to review because of the large body of evidence and the many questions [the authors] were trying to address,” Dr. Graber said. Dr. Graber understands that people are upset about the limited number of studies used to calculate the diagnostic error rate, but he said there just aren’t very many studies available, especially high-quality studies.

“I thought this study did an honest job in saying ‘these are the studies we used, this is how we analyze the data, this is the number we came up with,’” Dr. Graber said. “It does point out the need for much more research in this area.” 

For his part, Dr. Graber was most concerned that his feedback about how to improve diagnosis in the ED wasn’t incorporated into the revisions. He wished the authors expanded that to include more actionable solutions. “At this point in the game, we know that there are many other ways to improve diagnosis that weren’t touched upon at all [in the report],” Dr. Graber said. “Things like getting second opinions, using resources for decision support, getting better patient engagement, these are all things that have been proposed to improve diagnosis in other settings and would very likely be very productive in the emergency department.”

In addition to the reviewers and experts listed on the final report, AHRQ’s process includes a period of public review for the draft version of the systematic review. The writing team is obligated to respond to all comments submitted during public review, and those comments are made public three months after the report is published. Because of the unusually strong public reaction to this report, Craig Umscheid, MD, MS, Director of AHRQ’s Evidence-based Practice Center Division, told ACEP Now that AHRQ is looking to make those available before its normal three-month process to increase transparency.

Hardeep Singh, MD, MPH, a Professor of Medicine at Baylor College of Medicine, is a quality and safety researcher who has studied diagnostic error rates since 2005. He understands how challenging it is to analyze the limited data and measure preventability and attribution in the emergency care setting. Dr. Singh reviewed the draft in March 2022 and submitted his concerns via the public comment process. His public comments  spotlighted substantial scientific concerns and methodological flaws. “It is full of ‘convenient’ extrapolations and cherry picking,” he wrote, and he recommended substantial revisions to ensure that future research and quality improvement efforts don’t focus on potentially incorrect epidemiology and solutions based on this cherry-picked data.

Dr. Singh was disappointed when he saw the report was published without revisions to address the key objections he and other academics raised during public comment period.

“We, as researchers, have to be more responsible,” Dr. Singh said.  “With that comes accountability for making estimates of errors and harm using extremely rigorous science. We need to be able to protect the scientific integrity of the work we do.”

Dr. Singh made clear that he does think diagnostic errors are a significant problem, both inside and outside of the emergency department. “But when numbers are estimated with flawed data, you lose trust. And making estimates from studies that didn’t even measure diagnostic error and sensationalizing them is a fundamental problem that won’t help our case for improvements.” he said.   

Another diagnostic error researcher who spoke with ACEP Now on the condition of anonymity also submitted public comments for the AHRQ report. The researcher explained that diagnostic error research is very difficult to quantify, especially when considering preventability and causality of patient deaths, so it’s important to message the results with caution. They felt the body of the report explained its methodology and reasoning well, but that context was lost in a conclusion lacking nuance. “The main message is way too strong for the data that they have, the work that is out there,” the researcher said. “A much better conclusion would be that there’s a lot more research to be done because the numbers are not known.”

Nuance is important, especially when drawing conclusions about U.S. emergency departments by extrapolating data from studies done in Switzerland, where emergency medicine is not recognized as a specialty, or from Spain, where residency training in emergency medicine began just over a decade ago.

Jeremiah Schuur, MD, MHS, a patient safety expert and adjunct professor of emergency medicine at the Alpert Medical School of Brown University, said that the report “doesn’t pass the sniff test.” He said if the report’s data is true, then emergency physicians would miss one diagnosis every shift. “As someone who has overseen Mortality & Morbidity conferences at multiple facilities, that is not the experience that I’ve had,” he said.

When discussing the headline grabbing-numbers about potentially deadly errors, Dr. Schuur suggested it could have been framed differently. “When you have very imprecise results, it’s often better to frame them in that way and talk about [mortality rate] in a range of possibilities.”

“It’s frustrating that the process did not get the authorship team to modify their report,” Dr. Schuur said.

Dr. Panthagani, the Yale emergency medicine resident, heard about the study and its subsequent media coverage and decided to dig into the numbers herself. She echoed the same frustration as Dr. Schuur. “Bad data just really bothers me,” Dr. Panthagani said. The goal of her popular Twitter thread was to explain the holes in the data and make it clear that the public shouldn’t be afraid to come to the ED for emergency care. Dr. Panthagani took issue with the portion of the report in which the authors calculate the range of fatal medical errors in the ED. The report’s abstract presents the fatal medical error rate at 0.2 percent, but the body of the report explains that this number is derived from one small study with only one death, and the uncertainty of that rate actually ranges from 0.005 percent to 1.1 percent, or 6,500 to 1.4 million deaths.

The authors of the report call this confidence interval “implausibly wide” and define a new one. What was presented to the public was their own, much smaller “plausible range” estimate of fatal medical errors — 0.1 to 0.4 percent.

In response to questions about methodology, the authors sent ACEP Now a statement indicating they are working on a response to the most frequently asked questions about the report. The study authors will publish this FAQ in the coming weeks. “We have received a number of questions and concerns about the provenance, methods, or conclusions in the 744-page report. Some of these are fair critiques that deserve a thorough response; others are based on misunderstandings that can be answered through clarifying our approach and findings.”

Dr. Umscheid said the AHRQ staff is closely monitoring all feedback, including social media reactions such as Dr. Panthagani’s. AHRQ reviews the private and public comments submitted, along with the author response to those comments, prior to publication. But this level of post-publication controversy is unusual for AHRQ.  Dr. Umscheid said he hopes is that they can “continue the discourse around this important topic.”

“Nobody wants this type of feedback,” Dr. Umscheid said. “We’d much rather have feedback where people are not only interested in the findings, but are engaged with the findings and want to take the findings and move forward. I’ll speak for myself in saying it’s important for me and important for us to listen to the feedback.”