COVID-19 mRNA vaccines saved lives by reducing risks of infection and severe COVID-19

Large randomized clinical trials (RCT) showed that COVID-19 vaccines were safe and effective against COVID-19, a disease that caused more than 6 million deaths worldwide as of May 2023. After the first year of mass vaccination worldwide, mathematical models suggested it helped avert an extra 20 million COVID-19 deaths globally^[1].

However, posts on social media appearing in May 2023 claimed that COVID-19 vaccines using mRNA technology “didn’t actually save any lives” because their protective effect against COVID-19 deaths was “offset by other kinds of deaths attributable to the vax itself”. Such a claim lends readers the impression that, in the grand scheme of things, mRNA vaccines provided no benefit at all.

The research behind the claim

This claim originated from an article by The Epoch Times that reported on a study published by Benn et al.^[2]. In this study, the authors analyzed RCT data for four widely distributed vaccines: Pfizer-BioNTech, Moderna, AstraZeneca and Johnson & Johnson. Both the Pfizer and Moderna vaccines used mRNA technology, while the others used an adenovirus (AV) platform.

The authors compared overall mortality and non-COVID, non-accident mortality between mRNA and AV vaccine clinical trials. They reported that mRNA vaccines didn’t reduce overall mortality, in contrast with AV vaccines. This result forms the basis for the claim that mRNA vaccines didn’t save any lives.

Benn et al. also reported that there was an increase in non-COVID mortality in the mRNA vaccine clinical trials. The authors suggested that this increase counteracted the reduction in COVID-19 mortality, thus mRNA vaccines produced no overall benefit on balance. By contrast, the authors argued that AV vaccines would reduce both COVID-19 and non-COVID mortality, making them a better option overall.

The study contained small numbers of deaths making it vulnerable to statistical flukes

However, the study has several limitations. Posts and articles that reported the paper’s results without detailing the caveats thus misled their readers into giving too much credit to the study’s message.

Health Feedback reached out to several experts regarding the study, who pointed out that it contained several important limitations that render the study’s conclusions unreliable. [Read scientists’ comments in full below.]

First, Benn et al. used RCTs with a small number of deaths, whether from COVID-19 or other causes. Together, the mRNA vaccine (Pfizer-BioNTech and Moderna) RCTs reported 61 deaths out of 74,193 participants (0.0008%), while the AV vaccine (AstraZeneca and Johnson & Johnson) trials reported 46 deaths out of 122,164 participants (0.0003%). When assessing the risk of death, a small number of events is problematic because it leaves the results highly vulnerable to statistical flukes.

Karina Top, a professor in the division of infectious diseases of Dalhousie University, explained that these small numbers of deaths “translated into wide confidence intervals around the point estimates indicating lack of precision in the results”.

Abram Wagner, a professor of epidemiology and global public health at Michigan University, reached the same conclusion. “Together, these studies included 74,193 participants in mRNA vaccine trials, and there were only 61 deaths across vaccinated and placebo groups. Essentially there is not enough statistical power to make any conclusions from these data”, he said.

Statistical power is the ability of an analysis to adequately detect an effect where there is indeed one. In this case, a lack of statistical power due to a small number of deaths means that there’s a high risk of failing to detect mRNA vaccines’ effect on overall mortality when such an effect actually exists. In other words, data on mortality in the mRNA clinical trials reported in Benn et al. are actually inconclusive.

The fact that there aren’t enough deaths to properly assess the effect of vaccination on all-cause mortality isn’t surprising. As Benn et al. acknowledged, the initial clinical trials that led to vaccines’ authorizations primarily focused on preventing symptomatic COVID-19. For example, death from COVID-19 isn’t among the original primary outcomes monitored during the Pfizer, Moderna, or AstraZeneca clinical trials.

Once the vaccines’ efficacy against symptomatic COVID-19 was demonstrated, participants from the placebo groups were given the choice to receive the vaccine. Extending the duration of the trial would have left people from the placebo group at risk of getting COVID-19 when it could have been avoided.

David Boulware, a professor of medicine at the University of Minnesota, told Health Feedback that “What the authors are indirectly advocating for is unethical—to allow more harm to accrue to study participants after a clear benefit is known”.

The study examined trials that included very different populations, making reliable comparisons difficult

Another limitation of the study is that it compared the mortality in mRNA vaccines RCTs to the mortality in AV vaccines RCTs. However, this comparison is problematic because these RCTs were conducted on different populations and employed different protocols. These protocols involve various measures taken to reduce the risk of bias.

For example, randomization ensures that the demographic or medical characteristics of participants are evenly distributed between the control and the test groups^[3].

Because the groups within the same RCT are subjected to the same protocol, they can be reliably compared with each other. However, Benn et al. compared trials performed independently of each other in different locations and ignored the fact that each trial can be operated differently from another, Boulware pointed out. Ultimately, this is a comparison of apples and oranges.

One of the important differences is the number of people in mRNA or AV RCTs. Prof. Top explained that “mRNA population in particular was much smaller (<80,000) than the combined adenoviral vector vaccine trials (>120,000), which would lead to lower statistical power to identify a difference with mRNA vaccines”.

Another key difference is that mRNA and AV RCTs took place in different parts of the world, with different populations, background health, and access to healthcare.

“The adenovirus vaccine trials that were run mostly in low- and middle-income countries had an overall COVID case fatality rate of ~2%. The case fatality rate in the mRNA vaccine trials, run mostly in high-income countries (i.e. USA), was 0.3% based on the population enrolled and access to advanced healthcare“, said Boulware. As a result, the baseline overall mortality rate is different between the population in the mRNA RCTs and the one in the AV RCTs.

Another problem is that each RCT has its own protocol and its own way of assessing and ruling the cause of death. “How each trial may have classified deaths and the resources available were different. [Therefore], saying a vaccine protects against non-COVID deaths ignores the realities of problems in [correct] classification of cause of death,” Boulware explained.

Additional differences may also weaken the authors’ conclusions. For example, one of the AV vaccine RCTs used a meningitis vaccine for their control group, while all the others used a saline placebo. In addition, while they used the same technology, there are still marked differences between the AstraZeneca and Johnson & Johnson vaccines. For example, the former used a chimpanzee adenovirus and required two injections, while the latter used human adenovirus and required a single injection.

Top concluded that “[all] these factors could contribute to differences between vaccine types but were not controlled for in the analysis”. While Benn et al. acknowledged several of these shortcomings in their paper, it didn’t alter their overall conclusions. Therefore, online posts and articles claiming that the study showed mRNA COVID-19 vaccines didn’t save lives also didn’t explain how the paper’s limitations weaken its conclusions either, thus misleading readers into believing that the mRNA vaccines provided no benefit after all.

Real-world studies contradict claims of increased mortality

Finally, other studies contradicted the results from Benn et al.. While observational studies of the real-world vaccinated population carry higher risks of bias than RCTs, real-world data may have the upper hand when it comes to better representing the actual population that received the vaccine.

A study in people aged 12 to 29 in England found no differences in all-cause mortality risks between mRNA and non-mRNA vaccines, partially contradicting the results from Benn et al.^[4]. Similarly, a study on 11 million people in the U.S. between December 2020 and July 2021 found that the mRNA vaccines and the AV vaccine from Johnson & Johnson all reduced the risk of non-COVID-19 deaths^[5].

A study in those aged 70 and above in Norway also found that mRNA vaccines reduced all-cause mortality^[6].

Another study investigated all-cause mortality in Hungary. This research was also cited by Benn et al., who suggested that it showed a beneficial effect on overall mortality of AV vaccines “in comparison with Pfizer and Moderna mRNA vaccines”. However, the Hungarian paper actually reported that both AV and mRNA vaccines reduced the risk of all-cause mortality^[7].

Conclusion

In summary, initial RCTs showed that mRNA and AV COVID-19 vaccines are safe and effective at preventing severe disease. But they weren’t designed to investigate the vaccines’ effect on all-cause mortality. Consequently, using their data to arrive at conclusions about all-cause mortality involves navigating a minefield of problems. A study by Benn et al. attempted to do so by combining and comparing these RCTs. However, the researchers didn’t account for the fact that the RCTs were designed and conducted in different ways and didn’t have adequate statistical power to assess mRNA and AV vaccines’ effects on non-COVID mortality. Therefore, this study provides an unreliable basis for the claim that mRNA vaccines didn’t save lives. Furthermore, real-world data including millions of people counter this conclusion, showing instead that COVID-19 vaccines reduce mortality by reducing severe disease and death.

The fundamental claim is that the adenovirus vector vaccines (J&J, Astra-Zeneca, etc) have an all-cause mortality benefit whereas the mRNA vaccines did not in the clinical trials.

There are numerous problems with this conclusion. First, the mortality in the adenovirus vaccine trials that were run mostly in low and middle income countries had an overall COVID case fatality rate of ~2%. The case fatality rate in the mRNA vaccine trials, run mostly in high-income countries (i.e. USA), was 0.3% based on the population enrolled and access to advanced healthcare. Deaths were too few to show any reduction.

Second, the authors are cross-comparing trials in different locations and ignoring that. Each of the trials was run independently, and how trials operate can be different. Having run numerous clinical trials over the years, I know that determining the “cause of death” is often actually really difficult. How each of these trials operated and exactly ascertained cause of death OR attributed cause of death may be different.

For example, is a COVID-related death only if it’s directly related to COVID? What if someone develops a complication, such as a deep vein thrombosis (i.e. blood clot) and then dies of a pulmonary embolism (blood clot that goes to the lung)? Is that a COVID death? Or a non-COVID-death due to a blood clot? What if someone doesn’t know they have a blood clot, and they just die, thereby cause unknown? How each trial may have classified deaths and the resources available were different. Therefore, saying a vaccine protects against non-COVID deaths ignores the realities of problems in [correct] classification of cause of death. The biological plausibility is also a stretch.

This is armchair quarterbacking after the fact to pool data and then complain that the mRNA vaccines did not show a mortality impact because the trials did not run long enough to allow harm to occur to more people after the benefit of protection from COVID-19 was statistically clear. What the authors are indirectly advocating for is unethical: to allow more harm to accrue to study participants after a clear benefit is known.

What the authors may have preferred or are indirectly proposing: sure, the U.S. government could have waited another two to three months for the Pfizer and Moderna trials to run longer in order to show a mortality benefit (and also thereby withhold mRNA vaccines from the public, resulting in hundreds of thousands of additional COVID-19 deaths in early 2021). But that’s not ethical once the trial showed a conclusive benefit of protection from acquiring COVID-19 and developing severe COVID-19. That would have been horrible public policy and unethical.

Ultimately, there is a real-world truth. Just because someone can publish something does not make it true.The real-world data with millions of people does not match their analysis. One example from Israel shows this^[8].

Scientists have a responsibility to not just report data but to report what they believe is the truth.

In general, while the results are worthy of publication, there are important limitations that the authors highlight as well:

RCT sample sizes are small to detect rare outcomes like COVID mortality, and clinical trial participants are generally healthier than the overall population, so one expects lower COVID-related and overall mortality than the general population. These factors led to small numbers of deaths which translated into wide confidence intervals around the point estimates indicating lack of precision in the results.

The mRNA population in particular was much smaller (<80,000) than the combined adenoviral vector vaccine trials (>120,000) which would lead to lower statistical power to identify a difference with mRNA vaccines.

In addition, as the authors note, there were more COVID cases in mRNA trials but lower mortality versus adenoviral vector trials, suggesting significant differences in trial populations, settings, access to health care to manage COVID (and other diseases), differences in COVID rates, variants of concern, etc. There may also be differences in rates of other chronic diseases. All these factors could contribute to differences between vaccine types but were not controlled for in the analysis.

Population-based cohort studies are needed to assess overall mortality in the entire population, including groups excluded from clinical trials (e.g., pregnant individuals, people on immunosuppressive treatment, residents of skilled nursing facilities) which include individuals at highest risk of COVID mortality.

Watson et al. modeled deaths averted from COVID-19 vaccination in this recent paper^[1].

Since 2022, there have been over six million deaths from COVID-19 worldwide, including over one million in the U.S. mRNA vaccines, including those produced by Pfizer and Moderna, have been shown to be safe and effective, and have been administered to billions of people worldwide.

Benn and colleagues’ study on mortality used data from previously conducted randomized controlled trials. Together, these studies included 74,193 participants in mRNA vaccine trials, and there were only 61 deaths across vaccinated and placebo groups. Essentially there is not enough statistical power to make any conclusions from these data.

Of course, post-marketing surveillance of vaccines to assess safety, including deaths, is incredibly important. But it is scientifically inexcusable to make the comment that mRNA vaccines do not reduce overall mortality. The randomized controlled studies were not powered to make this claim. Moreover, requiring vaccine trials to formally register mortality as a statistical endpoint would substantially increase costs and delay the release of study findings. If this would have occurred during the COVID-19 pandemic, it is likely that many more individuals would have died.

Benn and colleagues’ study has a number of issues. It bases its claim off of an outcome – death – that only occurred in a handful of individuals. Statistically, they present many results, but if they had appropriately controlled for multiple statistical tests, their confidence intervals would have likely increased further, and they would have had less of a claim of the non-specific benefits of non-mRNA vaccines.

REFERENCES

• 1 – Watson et al. (2022) Global impact of the first year of COVID-19 vaccination: a mathematical modelling study. The Lancet Infectious Diseases.
• 2 – Benn et al. (2023) Randomized clinical trials of COVID-19 vaccines: Do adenovirus-vector vaccines have beneficial non-specific effects? iScience.
• 3 – Kang et al. (2008) Issues in Outcomes Research: An Overview of Randomization Techniques for Clinical Trials. Journal of Athletic Training
• 4 – Nafilyan et al. (2023) Risk of death following COVID-19 vaccination or positive SARS-CoV-2 test in young people in England. Nature communications.
• 5 – Xu e al. (2021) COVID-19 Vaccination and Non–COVID-19 Mortality Risk — Seven Integrated Health Care Organizations, United States, December 14, 2020–July 31, 2021. Morbidity and Mortality Weekly Report.
• 6 – Lopez-Doriga Ruiz et al. (2023) Short-term safety of COVID-19 mRNA vaccines with respect to all-cause mortality in the older population in Norway. Vaccine.
• 7 – Pálinkás & Sándor (2022) Effectiveness of COVID-19 Vaccination in Preventing All-Cause Mortality among Adults during the Third Wave of the Epidemic in Hungary: Nationwide Retrospective Cohort Study. Vaccines (Basel).
• 8 – Haas et al. (2021) Impact and effectiveness of mRNA BNT162b2 vaccine against SARS-CoV-2 infections and COVID-19 cases, hospitalisations, and deaths following a nationwide vaccination campaign in Israel: an observational study using national surveillance data. The Lancet.