Every generation of emergency physicians can tell a whiplash-inducing story of a medical reversal from their training, whether it be measuring pulmonary artery wedge pressures, the rise and fall of drotrecogin alfa, early goal-directed therapy for sepsis, or high-dose steroids for spinal cord trauma. Many of these therapies were diffused widely across practice before falling out of favor, and while each individual instance offers specific lessons, the larger question must be: why does this keep happening? 

The answer is, unfortunately, obvious, and subsequently disheartening for the future of emergency medicine (EM). There is a foundational issue with the quality of the evidence that informs clinical practice. We have seen recent articles in the Annals of Emergency Medicine describing trials in EM as “fragile,” but a new study evaluates the evidence base in its totality.¹ 

Producing what the authors term an “umbrella review,” this study sought out the top level of the evidence-based medicine pyramid for EM: systematic reviews and meta-analyses.²  The authors identified 431 eligible meta-analyses in EM in their search, comprised of 3,129 individual studies, the majority of which were randomized controlled trials. The authors rigorously evaluated each meta-analysis to determine whether there were sources of potential systematic bias affecting the reliability of the outcomes measured. 

First, each meta-analysis was evaluated for signs of publication bias. Publication bias colors the evidence base as consequence of suppression of a subset of trial results. This may happen as an inadvertent effect of academic publishing, as null results may be less appealing to medical journals and have a lower likelihood of acceptance. However, publication bias may also relate to an intentional discarding of negative results by those with professional or financial conflicts of interest. In either instance, the use techniques such as funnel plots and Egger’s regression may reveal signs of potential bias. 

The authors then exercised a more novel approach to the presence of bias: a test for excess significance. In a world where randomness ought to rule the day, it is a potential signal of threats to the internal validity of included studies if a disproportionate number reach statistical significance. Given a certain power to detect significant differences, there is an objective statistical likelihood for a pool of studies to each reach that threshold for significance based on their sample sizes. In effect, these authors have formalized “too good to be true” into an evaluation for possible bias. 

Lastly, the authors took those meta-analyses surviving the first two steps and looked at whether they had performed Grading of Recommendations, Assessment, Development and Evaluations (GRADE) assessments. These assessments are best-practice components to evaluate the strength of the individual studies included. 

Out of all these meta-analyses, the authors found 152 which were statistically significant and favored an experimental intervention. Unfortunately, 95 of these were found to be potentially unreliable after assessment for bias. From the remaining 57, there remained only 12 when the authors excluded further meta-analyses which failed to include any robust RCTs with at least 80 percent power to detect a valid effect size. Ultimately, only 2.8 percent of the top-level evidence in EM is reliable. 

But, we’re not yet finished. 

In one additional analysis, the authors then repeated the meta-analyses for each of those 12 but excluded the low-quality, low-power trials. From this original 12, only four remained with sufficient effect size and trial quality to remain statistically significant. Thus, of this summary of the highest-level evidence base, there are just four elements of EM we can likely be confident are immune to medical reversal.  

Without further ado, the bedrock of EM: 

• Putting a senior doctor in triage  reduces the likelihood that a patient will leave without being seen. 

• Clopidogrel pre-treatment versus no clopidogrel pre-treatment prior to percutaneous coronary intervention for acute coronary syndrome reduces subsequent major coronary events. 
• Use of straight-needle venipuncture versus an intravenous cannula start for blood draws reduces the rate of hemolysis. 
• Glucocorticoid treatment plus usual care, versus usual care alone, for croup reduces the rate of return visits. 

Clearly, it is not the case that all the rest of our current practice of emergency medicine is wrong. Hearkening back to the original problem and question, it ought to be clear there simply isn’t strong enough evidence backing most practice to ensure it cannot be overturned in the future. 

What little solace may be offered is only of the refrain, “misery loves company.”  A review in a similar vein was performed looking at the professional guidelines issued by the American College of Cardiology/American Heart Association and the European Society of Cardiology.³ Across 51 guidelines covering such areas as coronary artery disease, heart failure, and electrophysiology, only 8.5 percent of current recommendations in the ACC/AHA guidelines could be classified as having the top level of evidence. In a longitudinal analysis, this had actually dropped from 11.5 percent back in 2009, despite the growing awareness of risks of medical reversal. 

Finally, a ruefully unsatisfying bit comes from an evaluation of The Joint Commission on Accreditation of Healthcare Organizations measures.⁴ The Joint Commission, functioning in the U.S. as a bit of a de facto quality-enforcement body, generally requires measure compliance as a result of the financial impact associated with potentially losing accreditation. The Joint Commission standards reports, also known as R3, “requirement, rationale and reference,” describe the evidence in support of their actionable standards. Breaking down the 20 actionable standards with 76 distinct components, the authors found only six were completely supported by cited references. Upon evaluating the quality of the evidence, the authors found only one of those six supported by GRADE level B evidence, while the remainder were GRADE level D: “very low”. While these standards may indeed reflect high-quality medical practice and improved outcomes for patients, it is clear compliance may also ultimately represent unsupported, low-value investment. 

In summary, much of our current clinical practice remains prone to medical reversal and the guidelines upon which we base our practice rely on low-quality evidence, as do the standard measures enforced for hospital accreditation. While this may seem cause for despair, the better lens through which to view these limitations is humility. While emergency physicians continue to work to provide the best care possible for patients, we must recognize and embrace that change is inevitable. 

 References 

1. Brown J, Lane A, Cooper C, Vassar M. The results of randomized controlled trials in emergency medicine are frequently fragile. Ann Emerg Med. 2019;73(6):565-576. 
2. Parish AJ, Yuan DMK, Raggi JR, Omotoso OO, West JR, Ioannidis JPA. An umbrella review of effect size, bias, and power across meta‐analyses in emergency medicine. Academic Emergency Medicine. 2021;28(12):1379-1388. 
3. Fanaroff AC, Califf RM, Windecker S, Smith SC, Lopes RD. Levels of evidence supporting American College of Cardiology/American Heart Association and European Society of Cardiology guidelines, 2008-2018. JAMA. 2019;321(11):1069. 
4. Ibrahim SA, Reynolds KA, Poon E, Alam M. The evidence base for US Joint Commission hospital accreditation standards: cross sectional study. BMJ. 2022:e063064. doi: 10.1136/bmj-2020-063064. Available online at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9215261. Accessed October 9, 2022.