At least as important as carving out the time to read new studies, it is important to develop the skills to critically appraise new studies. The last few years may not be different than any previous decade as far as the quantity of biased, half-baked, and simply fraudulent research published, but the lack of practiced skill in evaluation is on prominent display. While examples of bias across pandemic-related topics are not restricted to any single point-of-view, a recent Cochrane Review publication has yet again refreshed a persistent topic of disagreement: the efficacy of masks and other non-pharmaceutical interventions to prevent the spread of respiratory viruses.¹ In a world of mostly transient obsessions with hydroxychloroquine, ivermectin, and various and sundry vitamins, the mask provocations remain.

Use of the PICO Structure

The best approach to understanding the outcome of this Cochrane Review is first to understand the methods of a systematic review. Using the “population, intervention, comparison, and outcomes” (PICO) structure, a research question is posed. Certain types of evidence are then chosen for inclusion in the review, ranging across observational, retrospective, prospective, or randomized controlled trials (RCTs). Specific medical literature databases are searched for articles meeting criteria, and distilled down into the a relevant cohort. Finally, the results are pooled and analyzed together to generate an estimate of treatment effect, ostensibly with greater certainty and across a broader population than any individual study.

At its foundation, therefore, the inputs determine the outputs. The ultimate conclusions are shaped by rules governing which studies are included, and, by definition, excluded. With all the levers available for manipulation, it becomes clear the scales can be tilted to produce objective results aligned with a particular chosen conclusion.

Assessment of Biases

The Cochrane Review methodology does not prevent erroneous results, rather, it governs primarily the structure of the analysis and reporting. Each included study is required to undergo an assessment of biases. These biases include assessments of threats to balanced enrollment, outcomes assessment, and completeness of results. Likewise, each study included is formally tabulated to describe its relationship to the initial PICO question, and the quality of its measurement of the effects. Unfortunately, these formal processes still do not prevent inclusion and pooling of largely irrelevant results from poorly conducted trials.

For example, a Cochrane Review of parachutes to prevent death from jumping from an aircraft could be proposed. There is a great deal of observational evidence of poor outcomes absent parachute use, but little in the way of randomized-controlled trials. However, a structured search restricted to RCTs alone would uncover one such trial.² The risk of bias assessment would identify concerns regarding some elements in this trial. However, because this RCT of parachute use to prevent death failed to show a difference between groups, this would become the unavoidable output of such a Review. This is, naturally, akin to madness, and similar to the flaws misleading those who amplify this most recent review concerning masks.

To determine the beneficial effect of a protective intervention, each group should have equal exposure to potential harms. The experimental group should deploy the intervention with perfect adherence, and deploy the intervention in all settings for which an exposure event may occur. In doing so, any adverse outcome would solely result from lack of intervention efficacy.

Among the trials of mask wearing included in this Cochrane Review, none remotely resemble this standard. For example, an included trial examined mask use while in tents used by pilgrims during the Hajj.³ In the intervention group, self-reported daily mask use was a mere 25 percent, while the control group reported daily mask use of 14 percent. Another trial recruited students in a University of Michigan residence hall, from which staff observed an average of 0.0007 participants properly wearing a mask for each hour of observation.⁴ Three other trials described use of masks within households of patients with influenza-like illness, with adherence to mask use measured in a handful of hours per day, if at all.

Similarly problematic issues arise with respect to trials comparing the effectiveness of medical masks and N95 respirators in health care settings. A few of these trials, performed primarily in Chinese hospitals during influenza seasons, showed dose-dependent positive effects favoring N95 respiratory use when comparing continuous, intermittent, and no respiratory use in hospitals.^5,6 In contrast, a more prominently published trial failed to show any advantage.⁷ In the trial failing to demonstrate any difference, however, N95 masks were worn solely when caring for patients with febrile respiratory illness. The remaining time, whether in staff stations, with other patients, or in the community, trial participants were unmasked. Clearly, an intervention is not reliably tested if exposure events occur outside the time in which the intervention is deployed.

However, the pooled results cannot reflect anything other than the included trials, and it becomes inevitable the authors of this Cochrane Review report a failure to demonstrate advantages to mask use. The vast majority of the amplification of this review focuses on this finding, including by the lead author to the lay media.⁸ This runs in stark contrast to the actual conclusions of the review, which lead off with a very sensible: “The high risk of bias in the trials, variation in outcome measurement, and relatively low adherence with the interventions during the studies hampers drawing firm conclusions.”

In most cases where RCT evidence is too weak upon which to draw any firm conclusions, the prudent course of action is to incorporate evidence from further down the evidence pyramid. Retrospective cohorts, case-control studies, before-and-after reports, and other quasi-experimental designs may have additional susceptibilities to bias, but frequently remain the best option to inform medical and policy decision-making when the RCT evidence is unhelpful.

An informative resource to this effect is, actually, the Cochrane Review on the same topic published in 2011.⁹ Curiously, this version was led by much the same authorship team as the present version, but arriving at quite different conclusions. The primary difference, other than including studies only up through 2010, was the inclusion of any comparative design in which some attempt was made to control for confounding. This included case-control, before-and-after, and other quasi-experimental designs.

The results are starkly different. These additional studies include those evaluating use of simple surgical masks comprising over 3,000 participants during the initial Severe Acute Respiratory Syndrome (SARS) epidemic in 2003. In these case-control studies, those wearing a mask were one-third as likely as controls to contract SARS. Fewer case-control studies assessed the use of N95 respirators, but the risk for contracting SARS was halved yet again in those who regularly wore N95 masks when caring for infected cases. In contrast to the present-day apparent inability to draw conclusions regarding masks, the pre-pandemic view of these same authors was: “Simple and low-cost interventions would be useful for reducing transmission of epidemic respiratory viruses.”

These previously included studies also require the same critical eye as those trials included in the most recent review, and possess their own limitations and generalizability issues. However, it is fallacious and unserious to suggest a lack of relevance to the present day, or to imply the most recent Cochrane Review has any bearing on the question of whether masks “work.” Rather than addressing mask efficacy, which is still best informed by the those original studies, the more recent trials raise a different question regarding the effectiveness of mask recommendations. Masking certainly reduces viral transmission, but in order to develop relevant infection control effects at a population level, other factors relating to mask use, uptake, and other public health measures require consideration. Regrettably, in self-reinforcing fashion, casting doubt upon the efficacy of masks ultimately diminishes their population-level effectiveness, a vicious cycle favoring those who decry the usefulness of masks.

As the pandemic winds down, more individuals have durable protection against severe disease and community viral prevalence is subsiding. These factors make routine mask use dramatically less important than early in the pandemic. However, preventing mischaracterization of these data, and other specious representations of studies, remains critical preparation for future public health activities.

Dr. Radecki is an emergency physician and informatician with Christchurch Hospital in Christchurch, New Zealand. He is the Annals of Emergency Medicine podcast co-host and Journal Club editor and can be found on Twitter @emlitofnote.

References

1. Jefferson T, Del Mar CB , Liz Dooley L. Physical interventions to interrupt or reduce the spread of respiratory viruses. Cochrane Database Syst Rev. 2020;11(11):CD006207. doi: 10.1002/14651858.CD006207.pub5.
2. Yeh, RW. Parachute use to prevent death and major trauma when jumping from aircraft: randomized controlled trial. BMJ. 2018;363:k5094
3. Alfelali M, Haworth EA, Barasheed O, et al. Facemask against viral respiratory infections among Hajj pilgrims: A challenging cluster-randomized trial. PLoS ONE. 2020;15(10):e0240287.
4. Aiello AE, Perez V, Coulborn RM, Davis BM, Uddin M, Monto AS. Facemasks, hand hygiene, and influenza among young adults: a randomized intervention trial. PLoS ONE. 2012;7(1):e29744.
5. MacIntyre CR, Wang Q, Cauchemez S, et al. A cluster randomized clinical trial comparing fit-tested and non-fit-tested N95 respirators to medical masks to prevent respiratory virus infection in health care workers: RCT of face masks in health workers. Influenza Other Respir Viruses. 2011;5(3):170-179.
6. MacIntyre CR, Wang Q, Seale H, et al. A randomized clinical trial of three options for n95 respirators and medical masks in health workers. Am J Respir Crit Care Med. 2013;187(9):960-966.
7. Loeb M, Dafoe N, Mahony J, et al. Surgical mask vs n95 respirator for preventing influenza among health care workers: a randomized trial. JAMA. 2009;302(17):1865.
8. Henegan C, Jefferson T. Do mask mandates work? A look at the evidence. The Spectator website. https://www.spectator.co.uk/article/do-mask-mandates-work/ Published February 3, 2023. Accessed March 11, 2023.
9. Jefferson T, Dooley L, Ferroni E, et al. Physical interventions to interrupt or reduce the spread of respiratory viruses. Cochrane Database Syst Rev. 2011;2011(7):CD006207. doi: 10.1002/14651858.CD006207.pub4.