Multiple studies show that face masks reduce the spread of COVID-19; a Cochrane review doesn’t demonstrate otherwise

A review published by the Cochrane Library on 30 January 2023 triggered renewed claims that face masks are ineffective at reducing the spread of SARS-CoV-2 and other respiratory infections. This review, which analyzed mask effectiveness data pooled from multiple studies, reported that wearing a face mask made “little or no difference” in the number of influenza and COVID-19 cases^[1].

Many social media users shared these results as evidence that masks don’t work. Some also criticized the U.S. Centers for Disease Control and Prevention (CDC) for recommending mask usage during the COVID-19 pandemic that supposedly lacked scientific support (examples here, here, and here). These claims aren’t new, as multiple reviews by Health Feedback documented.

Robert Malone, a scientist who previously spread misinformation about COVID-19 vaccines, was among the first ones to share the results of the Cochrane review in such a manner. In a Substack article later republished by the Brownstone Institute, Malone claimed that the Cochrane review “settled” the debate on mask-wearing and pointed to potential effects of masks on “health, childhood development, speech development”.

Similarly, other websites like Reason.com, newspapers like Chicago Sun-Times, and media outlets like Fox News claimed that the CDC had “exaggerated” the evidence supporting mask mandates. These few articles together received over 85,000 interactions on Facebook, Instagram, and Twitter, according to social media analytics tool CrowdTangle. In addition, the Cochrane review received more than 50,000 interactions on Facebook and Instagram.

However, such an interpretation of the Cochrane review is misleading and unsupported by the review’s analysis because the limitations of the review prevent us from drawing reliable conclusions about the impact of mask-wearing in the real world, as we will explain below.

What did the Cochrane review do and what did it find?

The Cochrane review is the sixth version of a series of reviews evaluating the effectiveness of physical interventions in reducing the spread of viral respiratory infections. This type of intervention includes screening at entry ports, isolation, quarantine, physical distancing, personal protection, hand hygiene, face masks, glasses, and gargling. To do this, the authors conducted a meta-analysis, which is a statistical analysis that combines data from multiple scientific studies on a given topic.

The meta-analysis included 43 studies. Among them, 14 studies evaluated the impact of surgical masks and N95/P2 respirators, alone or combined with other interventions, on the number of cases of viral respiratory infections like COVID-19 and influenza.

In contrast to previous versions, the updated meta-analysis only included randomized controlled trials (RCTs), a type of study that evaluates the effectiveness of an intervention by comparing a group of people who received treatment to a control group who didn’t. In RCTs, all participants are randomly allocated into either the treatment or control group, for example by assigning them computer-generated random numbers. This process, called randomization, reduces biases due to how participants are assigned to each group.

The review concluded that “hand hygiene has a modest effect as a physical intervention to interrupt the spread of respiratory viruses”. In contrast, the analysis “did not show a clear reduction in respiratory viral infection with the use of medical/surgical masks”. There were also “no clear differences between the use of medical/surgical masks compared with N95/P2 respirators in healthcare workers when used in routine care to reduce respiratory viral infection”.

Many websites and social media users interpreted these results to mean that face masks are ineffective at reducing the spread of respiratory diseases, including COVID-19. However, this interpretation is misleading because, as the authors clearly state in the Discussion, “The variable quality of the studies hampers drawing any firm conclusions.”

The review only addressed part of the benefits of mask-wearing

Epidemiologists Raina MacIntyre, Abrar Ahmad Chughtai, David Fisman, and primary health care professor Trish Greenhalgh detailed the multiple limitations of the Cochrane review in an article for The Conversation. One of these limitations was that most trials included in the review “addressed only half of the question” on mask effectiveness:

“Face masks and respirators work in two ways: they protect the wearer from becoming infected and they prevent an infected wearer from spreading their germs to other people. Most RCTs in this Cochrane Review looked only at the former scenario, not the latter.” 

N95/P2 respirators are designed to prevent airborne infections, like COVID-19 and flu, by filtering infectious particles and preventing the wearer from breathing them in. By contrast, surgical face masks mainly work by physically blocking the release of infectious particles from infected individuals into the air (source control)^[2], as Health Feedback explained in an earlier review. However, most trials only tested mask effectiveness at preventing infection in the wearer, ignoring the potential benefit of face masks in source control.

Multiple methodological limitations in the Cochrane review cast doubt on the reliability of the results

While certain articles listed some of the review’s limitations, their headlines and overall narrative still conveyed the message that the review demonstrated that face masks are ineffective at reducing the spread of SARS-CoV-2. In fact, most of the articles focused on the alleged high quality of the review. This was the case in Malone’s article, which argued that “Cochran [sic] reviews are considered the gold standard for health care agencies and professionals”.

Certainly, meta-analyses can provide the best quality of evidence about the effectiveness of health interventions because they pool data from a large number of participants, offering much greater statistical power than individual studies. However, the capacity of a meta-analysis to answer a question with a high degree of confidence depends directly on the quality of the studies included in the analysis. For this reason, meta-analyses need to carefully account for any potential biases in the studies included, which can greatly influence the reliability of the overall results and conclusions.

Likewise, large and well-conducted RCTs are considered the gold standard for evaluating the effectiveness of an intervention. But this status doesn’t apply to any RCT, as not all RCTs render equally reliable results. RCTs can vary significantly in quality depending on the number of participants, the type of control group, and the way participants are randomized. Ideally, RCTs should also be double-blinded, which means that neither the participants nor the researchers know to which group each participant is allocated. Blinding makes it less likely that factors unrelated to the study, like the placebo effect or changes in behavior, influence the study’s results.

As the authors of the Cochrane review explained in the Discussion, the available RCTs evaluating mask effectiveness were of “variable quality”. For example, some of the trials lacked blinding, while others used unclear randomization methods or poorly defined outcomes to assess the impact of the intervention. Each of these factors increases the risk of bias, reducing the reliability of the meta-analysis’ conclusions. In addition, while some studies confirmed the type of infection by a laboratory test, many others relied on self-reporting to assess both mask-wearing and infection, further increasing the risk of bias.

As a result of these methodological issues, a large proportion of the RCTs included in the meta-analysis had a high or unclear risk of bias (Figure 1).

Figure 1. Authors’ judgments about the risk of different types of bias presented as percentages across all included trials. Based on their risk of each type of bias, all the studies were classified as low risk (green), unclear risk (yellow), or high risk (red) of bias.

Another important limitation of the review is that the RCTs included in the analysis evaluated the effect of face masks on several respiratory viruses in different populations and multiple settings with variable risks of transmission. This variability is already made clear at the beginning of the review:

“[The studies] took place in low-, middle-, and high-income countries worldwide: in hospitals, schools, homes, offices, childcare centres, and communities during non-epidemic influenza periods, the global H1N1 influenza pandemic in 2009, epidemic influenza seasons up to 2016, and during the COVID-19 pandemic.”

But pooling data from such heterogeneous studies together increases the likelihood that any effect gets diluted in the overall variability of the data.

Further complicating the question is the fact that most of the participants in the individual RCTs didn’t wear face masks consistently during the trial. Instead, most wore masks occasionally or in specific settings, like working places, university residences, or when in contact with people with respiratory infections.

Although study participants only wore masks part of the time, most studies didn’t measure mask-wearing compliance. The few studies that did record this reported a high variability ranging from 40 to 80%, which the authors defined as “relatively low numbers”. This lack of compliance is relevant because evidence indicates that the community benefit of face masks increases with the number of people who wear them.

In addition, the review mentioned other factors that might have influenced the results, including the quality and material of the mask, contamination by hands of saliva, and possible changes towards riskier behavior in people who wore masks due to an exaggerated sense of security.

Due to all these limitations, the authors’ confidence in the results was “low to moderate for the subjective outcomes related to respiratory illness” and “moderate for the more precisely defined laboratory-confirmed respiratory virus infection”. This means that, rather than demonstrating that masks are ineffective at reducing respiratory infections as claimed, the evidence was insufficient to conclude whether they have an effect. The authors highlighted the need for more, larger, and better-designed RCTs:

“In summary, more high-quality RCTs are needed to evaluate the most effective strategies to implement successful physical interventions in practice, both on a small scale and at a population level.”

In a viral opinion piece for the New York Times, journalist Bret Stephens didn’t take into account these limitations and claimed that, based on the review’s results, “when it comes to the population-level benefits of masking, the verdict is in: Mask mandates were a bust”. To further support this claim, the piece quoted several statements that the review’s first author, epidemiologist Tom Jefferson, made during a Substack interview with journalist Maryanne Demasi.

Stephen’s opinion piece not only misrepresented the review results as others did but also conflated the concepts of mask-wearing and mask mandates. These are very different things because the fact that a mandate is in place doesn’t necessarily mean that a majority of the people actually wear a mask or that they do it properly. Therefore, the effectiveness of mask mandates wouldn’t only depend on how effective face masks are but also on the level of compliance.

On 10 March 2023, the New York Times published another opinion piece by columnist Zeynep Tufekci. In it, Karla Soares-Weiser, the editor in chief of The Cochrane Library, apologized because the Cochrane review’s wording “was open to misinterpretation”. Soares-Weiser explained that claiming “‘masks don’t work” based on the Cochrane review’s results “is an inaccurate and misleading interpretation” and added that Jefferson’s statements in the interview were “not an accurate representation of what the review found”. The Cochrane Library released later that day an official statement, which can be read in full below.

Evidence from several high-quality studies strongly suggests that widespread and consistent mask-wearing limits the spread of SARS-CoV-2 in the real world

More than three years after Wuhan reported the first COVID-19 cases, the debate about whether and how much face masks limit the spread of SARS-CoV-2 is far from settled with by the Cochrane review. In fact, one of the main findings of the review is the need of better-quality data about the impact of face masks in different settings and circumstances.

For example, most experts agree that face masks were an essential part of the control measures during the COVID-19 pandemic. However, some question their effectiveness in periods when virus transmission is lower and, particularly in the cases of cloth masks, in the context of new and more infectious circulating variants.

The variable quality of the RCTs included in the Cochrane review mainly results from the difficulty of conducting robust randomized controlled trials on mask effectiveness. On the one hand, trials conducted during periods of low virus circulation—like many of the RCTs included in the Cochrane review—are generally underpowered because a low number of infections make the statistics less reliable. But on the other hand, preventing a group of people from using face masks amid a pandemic to evaluate the effectiveness of this measure at preventing infections would be considered unethical.

The problem is that low-quality or inadequately designed RCTs are less likely to detect an effect, which might be misinterpreted as evidence that masks don’t work. That is why some epidemiologists consider that a mix of well-conducted and more comparable RCTs and observational studies might produce more reliable results than pooling data from variable-quality RCTs addressing “quite different questions”.

Several robust observational studies conducted during the COVID-19 pandemic found that mask-wearing in community settings limited the spread of SARS-CoV-2 and reduced the number of infections. For example, one study conducted in nine regions in the U.S. state of California in 2021 found that people who consistently wore a face mask in public spaces were 50% less likely to test positive for SARS-CoV-2^[3]. While N95/P2 respirators offered the highest degree of protection, any mask worn consistently was more effective than no mask at reducing the likelihood of infection.

Earlier studies in Johnson County, Iowa, and Beijing households reported that when an uninfected person and an infected person both wore a face mask, the likelihood of the uninfected person to become infected fell by 50 to 79%^[4,5].

Another study, published in the New England Journal of Medicine in November 2022, showed that lifting mask requirements in school districts in Boston was associated with 50 additional COVID-19 cases per 1,000 students and staff during the 15 weeks after the statewide masking policy was rescinded^[6].

Unlike the Cochrane review, systematic reviews including studies beyond RCTs found that both N95 and surgical masks protected against SARS-CoV-2 infection in healthcare and community settings^[7,8].

Available data indicates that mask-wearing is more effective when combined with other control measures, such as physical distancing and frequent handwashing^[7,9]. These results underscore the importance of layering multiple prevention strategies to minimize the risk of infection, rather than dismissing a certain measure, like masks, simply because it doesn’t provide 100% protection (nirvana fallacy). Virologist Ian Mackay illustrated this concept in the graphic “Swiss Cheese Pandemic Defense” (Figure 2).

Figure 2. Graphic illustration showing how combining multiple, imperfect control measures can improve protection against COVID-19. Source: Ian Mackay.

Conclusion

Claims that face masks don’t work based on the results of the Cochrane review are unsupported and misrepresent the conclusions of the review, which couldn’t draw “any firm conclusions” on the effectiveness of mask-wearing at limiting the spread of respiratory infections due to methodological limitations. Many of those claims also mislead the readers by presenting the review as high-quality definite evidence without adequately acknowledging its limitations. While meta-analyses are indeed at the top of the quality evidence pyramid, their reliability depends on the quality of the individual studies they include, which is highly variable in the case of face masks.

A growing body of evidence from RCTs and observational studies suggests that consistent mask-wearing can effectively reduce the spread of respiratory viruses like SARS-CoV-2 in both healthcare and community settings. The extent to which community mask-wearing contributes to limiting the spread of different respiratory viruses and in different circumstances is still unclear. Hopefully, future well-designed studies will answer these questions. For the time being, face masks are another layer of protection in addition to vaccination, frequent handwashing, and physical distancing when the circulation of respiratory viruses is high.

Statement on ‘Physical interventions to interrupt or reduce the spread of respiratory viruses’ review

Karla Soares-Weiser, Editor-in-Chief of the Cochrane Library, on behalf of Cochrane:

“Many commentators have claimed that a recently-updated Cochrane Review shows that ‘masks don’t work’, which is an inaccurate and misleading interpretation.

It would be accurate to say that the review examined whether interventions to promote mask wearing help to slow the spread of respiratory viruses, and that the results were inconclusive. Given the limitations in the primary evidence, the review is not able to address the question of whether mask-wearing itself reduces people’s risk of contracting or spreading respiratory viruses. 

The review authors are clear on the limitations in the abstract: ‘The high risk of bias in the trials, variation in outcome measurement, and relatively low adherence with the interventions during the studies hampers drawing firm conclusions.’ Adherence in this context refers to the number of people who actually wore the provided masks when encouraged to do so as part of the intervention. For example, in the most heavily-weighted trial of interventions to promote community mask wearing, 42.3% of people in the intervention arm wore masks compared to 13.3% of those in the control arm. 

The original Plain Language Summary for this review stated that ‘We are uncertain whether wearing masks or N95/P2 respirators helps to slow the spread of respiratory viruses based on the studies we assessed.’ This wording was open to misinterpretation, for which we apologize. While scientific evidence is never immune to misinterpretation, we take responsibility for not making the wording clearer from the outset. We are engaging with the review authors with the aim of updating the Plain Language Summary and abstract to make clear that the review looked at whether interventions to promote mask wearing help to slow the spread of respiratory viruses.”

UPDATE (10 March 2023):

This review was updated to address two related opinion pieces published in the New York Times, one by Bret Stephens and a second one by Zeynep Tufekci. Three paragraphs were added at the end of the section discussing the limitations of the Cochrane review.

The update also included a statement by Karla Soares-Weiser, the editor in chief of The Cochrane Library, on behalf of the Cochrane Library, which was reproduced in full at the end of this review. This statement further supports our verdict and did not change it.

REFERENCES

• 1 – Jefferson et al. (2023) Physical interventions to interrupt or reduce the spread of respiratory viruses. Cochrane Library of Systematic Reviews.
• 2 – Fischer et al. (2020) Low-cost measurement of face mask efficacy for filtering expelled droplets during speech. Science Advances.
• 3 – Andrejko et al. (2022) Effectiveness of Face Mask or Respirator Use in Indoor Public Settings for Prevention of SARS-CoV-2 Infection — California, February–December 2021. Morbidity and Mortality Weekly Report.
• 4 – Riley et al. (2022) Mask Effectiveness for Preventing Secondary Cases of COVID-19, Johnson County, Iowa, USA. Emerging Infectious Diseases.
• 5 – Wang et al. (2020) Reduction of secondary transmission of SARS-CoV-2 in households by face mask use, disinfection and social distancing: a cohort study in Beijing, China. BMJ Global Health.
• 6 – Cowger et al. (2022) Lifting Universal Masking in Schools – Covid-19 Incidence among Students and Staff. New England Journal of Medicine.
• 7 – Talic et al. (2021) Effectiveness of public health measures in reducing the incidence of covid-19, SARS-CoV-2 transmission, and covid-19 mortality: systematic review and meta-analysis. BMJ.
• 8 – Chu et al. (2020) Physical distancing, face masks, and eye protection to prevent person-to-person transmission of SARS-CoV-2 and COVID-19: a systematic review and meta-analysis. The Lancet.
• 9 – Mello et al. (2022) Effectiveness of face masks in blocking the transmission of SARS-CoV-2: A preliminary evaluation of masks used by SARS-CoV-2-infected individuals. Plos ONE.