Assessing the effect of COVID-19 vaccines on mortality: a story of confounding factors and their role in COVID-19 misinformation

Introduction

COVID-19 vaccines have been instrumental in our fight against the pandemic and our return to a normal life, thanks to their ability to reduce the number of cases, hospitalizations, and mortality. Their effectiveness against severe COVID-19 and death was proven in randomized controlled trials (RCT) that involved tens of thousands of people^[1,2].

While RCTs are critical to our understanding of the vaccines’ efficacy, as they offer standardized and well-controlled test conditions and are therefore considered the gold standard for clinical assessment of a treatment, they don’t reflect the diversity and complexity in the real world.

Therefore, monitoring the impact of COVID-19 vaccination on mortality, even after the vaccines are approved or authorized for use in the public, remains an important part of vaccine surveillance. Consequently, official statistics on the number of COVID-19 deaths among vaccinated and unvaccinated individuals are publicly available in several countries, to allow researchers to study the vaccines’ real-world effectiveness.

Comparing COVID-19 mortality rates between vaccinated and unvaccinated people may seem like a simple matter of comparing the COVID-19 death counts between the two. But it’s actually more complicated than that. As we will explain below, vaccinated and unvaccinated populations can differ from each other in several important ways. Therefore, it’s critical to account for these differences when comparing mortality rates between the two.

COVID-19 death counts must be adjusted for vaccine coverage

One of the starkest differences between the vaccinated and unvaccinated population is their size. Failing to account for population size is a fundamental mistake that can introduce important statistical bias that invalidates a research finding.

To illustrate this, we can run a small mathematical thought experiment. Let’s imagine a population of 1,000 drivers with a 10% risk of dying in a car accident within a year. At the end of the year we would thus expect 100 deaths. We know that wearing a seat belt significantly reduces the risk of dying in a car crash, albeit not entirely. Let’s suppose that wearing the seat belt reduces the risk of death by half, down to 5%. If the entire population were wearing one, we would expect 50 deaths instead of 100.

Now, let’s consider a mixed situation where some people use a seat belt and some don’t. If 90% of the population—900 people—don’t wear a seat belt, the number of deaths in this group would be 10%, or 90 deaths. The remaining 100 people, who are using a seat belt, have a 5% death risk, so we would expect five deaths. In that case, deaths from non-seat belt wearers represent the majority of deaths in the total population.

Now, let’s invert the situation and consider that 90% of the population, 900 people, wear a seat belt, and the remaining 100 never do. The 100 individuals not wearing seat belts are still at a 10% risk of dying so we’d expect 10 deaths. The 900 people wearing a seat belt are at 5% risk of dying, which would translate into 45 deaths. In that case, we would see more deaths among seat belt wearers. Should we therefore conclude that seat belts suddenly stopped being effective, or worse, increased one’s risk of dying? Of course not. This is a mere mathematical consequence of the overrepresentation of seat belt wearers in the total population.

We can apply the same reasoning to vaccines. Even though pre-clinical and clinical data demonstrated the efficacy of COVID-19 vaccines in reducing the risk of severe disease, they don’t offer 100% protection, which is common to most vaccines against other diseases. So, COVID-19 deaths are still expected among a small proportion of vaccinated individuals.

As the COVID-19 vaccination campaign progresses, we expect the overall number of vaccinated people increases. In other words, the proportion of vaccinated people becomes larger than that of unvaccinated people. Figure 1 illustrates that the vaccine coverage rose to around 65% in the U.S. over time.

Figure 1. COVID-19 vaccine coverage of the U.S. population over time. Cumulative numbers of people who received at least two vaccine doses were obtained from the Centers for Disease Control and Prevention and expressed as a percentage of the total U.S. population

As the vaccine coverage increases, the number of COVID-19 deaths among vaccinated people naturally increases. As our seat belt example illustrated, this phenomenon is unrelated to the effectiveness of the safety intervention, specifically the vaccine in this case. Instead, it is a direct, mathematical consequence of the higher proportion of vaccinated individuals in the population.

Therefore, it is always necessary to ensure that we compare the proportion of COVID-19 deaths in each group, as opposed to the absolute number of COVID-19 deaths. One way to do so is by expressing the number of deaths per million vaccinated or unvaccinated persons.

Challenges in comparing COVID-19 mortality rates: the risk of confounding factors

There are also many other factors that influence the risk of death apart from vaccination status that researchers need to account for. These are known as confounding factors, which researchers need to control for. Confounding factors are variables that affect the outcome of an experiment, but aren’t the variables being studied in the experiment. If scientists don’t factor in the influence of confounding factors in their study, they may draw erroneous conclusions about causality.

One example of a potential confounding factor is access to healthcare, with one population that faces greater difficulty in accessing hospitals. It is more likely that the population without access difficulties to healthcare will experience a higher proportion of COVID-19 deaths than the other due to the lack of medical care, regardless of any possible effect of vaccines. In this case, access to healthcare is a confounding factor: it will influence the outcome of interest—death from COVID-19—independently from the vaccine’s effect, and will bias the observers’ conclusions if not taken into account.

In the following sections, we will explore some of the recurring confounding factors that have led to misinformation about vaccine effectiveness.

COVID-19 death counts must be adjusted for age

We know that the chances of surviving COVID-19 significantly decreases with age. As of 2 March 2022, people above 75 represented more than half of the COVID-19 deaths from COVID-19, according to the U.S. Centers for Disease Control and Prevention (CDC). Therefore, age is a possible confounding factor: if one population is significantly older than the other, it will alter that group’s risk of dying from COVID-19, independently of vaccination status.

Many countries initiated their vaccine rollout with vulnerable populations such as healthcare workers or older people. For instance, the U.S. Advisory Committee on Immunization Practice from the CDC recommended focusing the initial vaccine rollout on older people in nursing homes and assisted living facilities, followed by people above 75, people aged 64 to 75, and eventually everyone above 16. Later on, vaccination was extended to children aged 12 to 15, and finally, children aged 5 to 11.

As a result, vaccine coverage in older populations is higher than in younger populations. Consequently, the vaccinated population comprises a larger proportion of elderly individuals compared to the unvaccinated population. Although this difference tends to decrease as the vaccine coverage grows, data from the CDC shown in Figure 2 illustrate that this age group is still overrepresented.

Figure 2. Age distribution of the vaccinated population and of the total U.S. population as reported by the CDC as of 10 March 2022.

The dark red bars represent the percentage of people by age group who received at least one dose of the vaccine as of 6 March 2022. The gray bars represent the percentage of this age group in the total U.S. population. If the age distribution of the vaccinated population was similar to the age distribution of the total population, the gray and red bars of the same age group should be of roughly equal lengths. When the red bar from a given age group is longer, this means that this age group is overrepresented in the vaccinated population.

Therefore, if one were to observe an equal (or even higher) number of deaths from COVID-19 in the vaccinated population compared to the unvaccinated population, it would be erroneous to conclude that vaccines are ineffective. This is because the conclusion doesn’t account for the fact that vaccinated people are, on average, older and thus more likely to die from COVID-19, regardless of the vaccines’ effectiveness.

One way to mitigate confounding factors is by performing age adjustment. This statistical process adjusts the number of deaths by age group by a coefficient representing the proportion of this age group in the total country’s population. This would yield an age-adjusted total number of COVID-19 deaths in the vaccinated and unvaccinated populations, making comparisons more meaningful.

Data from the CDC illustrate clearly the impact of age adjustment (Figure 3). These data were collected from April 2021 to January 2022.

Figure 3. Effect of age adjustment on the incidence rate ratio (IRR) of COVID-19 mortality. Data were obtained from a CDC survey of COVID-19 deaths by vaccination status from April 2021 to January 2022. Weekly incidence rates are calculated as the number of deaths from COVID-19 among the vaccinated or unvaccinated population within a given week, expressed as the number of deaths per 100,000 individuals. Therefore, these numbers take into account the size difference of each population. The incidence rate ratio (IRR) is the ratio of the average incidence rate among unvaccinated divided by the average incidence rate among vaccinated. A IRR higher than one indicates that deaths occur at a higher frequency among unvaccinated. The crude IRR is obtained directly from the weekly incidence rates, whereas the age-adjusted IRR is obtained after a step of adjustment of the age distribution of both populations using the 2000 U.S. census standard population as a reference.

The incidence rate ratio (IRR) represents the excess occurrence of COVID-19 deaths among unvaccinated versus vaccinated. If the IRR is equal to one, there is no difference in COVID-19 mortality between the two groups. If the IRR is greater than one, it indicates that COVID-19 deaths occur at a higher rate among unvaccinated individuals.

Here it is important to note that data are also normalized by the size of the vaccinated and unvaccinated population. Therefore, there is no risk of confounding effects due to vaccine coverage, as explained earlier.

The gray column shows what the IRR would be without age adjustment. Even without this correction, the IRR is higher than one, showing that vaccinated people are dying from COVID-19 less than unvaccinated people.

When applying an age adjustment (blue column), we can see that the IRR increases further. This indicates two things: first, a crude observation, without taking into consideration that the vaccinated population tends to be older, underestimates the protective effect of vaccination. Second, vaccinated people are at lower risk of dying from COVID-19 than unvaccinated individuals, in fact 14 times lower in December 2021, according to the CDC.

Behavioral differences between vaccinated and unvaccinated create additional confounding factors

While the aforementioned confounding factors, namely magnitude of vaccine coverage and age distribution, are easier to account for in studies, the influence of other confounding factors can be harder to mitigate.

For instance, it is possible that vaccinated people may become more complacent and less observant of physical distancing and hygiene measures^[3,4], because of the immunity provided by vaccination, whereas unvaccinated people would not.

This would introduce a difference in behavior between both groups that has direct bearing on a person’s risk of getting COVID-19. Specifically, it would lead to a higher than expected number of COVID-19 cases in the vaccinated group, not because the vaccine is ineffective, but because the vaccinated group’s risk exposure changes as a result of their^[5,6] behavior.

Another type of behavioral bias is the potential difference in healthcare-seeking behavior between groups. We could hypothesize that people who chose to get vaccinated –and those who chose not to– have different views on health and different healthcare-seeking behavior. This might directly impact their risk of infection or death from the infection.

Not accounting for that bias can lead to ill-founded conclusions about vaccines, as Health Feedback previously explained. Again, it is difficult to fully address that confounding factor. However, certain specific clinical study designs such as the case-negative design, help reduce their impact of the study’s conclusions^[7,8].

How confounding factors fuel vaccine misinformation

As we explained earlier, a direct comparison of absolute COVID-19 deaths between vaccinated and unvaccinated groups is likely to be meaningless, due to the influence of multiple confounding factors. Indeed, the failure to understand how confounding factors could result in seemingly paradoxical numbers repeatedly produced inaccurate claims that COVID-19 vaccines were ineffective at preventing COVID-19 deaths, as Health Feedback documented on several occasions.

In each case, the claims suggested vaccine failure based on a higher or equal number of hospitalizations or deaths in vaccinated individuals compared to unvaccinated individuals. However, the authors of the claims didn’t take into account the extent of vaccine coverage in the population and the age discrepancy between vaccinated and unvaccinated groups.

As of 9 March 2022, vaccinated individuals are the majority in many countries. For instance, vaccinated people represent 65% of the total population in the U.S. and 85% in the U.K. It is thus possible that vaccinated people account for a high number of deaths regardless of vaccine effectiveness, as explained earlier.

In March 2022, another article committed the same mistake. It claimed that vaccinated individuals accounted for nine out of ten COVID-19 deaths in the U.K., according to official sources.

Specifically, a report by the U.K. Health Security Agency from 24 February 2022 indicated that deaths among unvaccinated people only accounted for 10% of the total number of deaths occurring within 60 days of a positive COVID-19 test. The fact that partially and fully vaccinated individuals accounted for the other 90% led the authors to conclude that vaccines didn’t work.

This reasoning is again flawed as the authors didn’t consider the two confounding factors we previously mentioned: age and vaccine coverage. The U.K. Health Security Agency even warned in bold type on page 37 of their report: “this raw data should not be used to estimate vaccine effectiveness”, adding that “The case rates in the vaccinated and unvaccinated populations are crude rates that do not take into account underlying statistical biases in the data”.

In November 2022, many posts implied that COVID-19 vaccines were either dangerous or ineffective at preventing deaths by sharing a headline from the Washington Post, stating that “vaccinated people now make up a majority of covid deaths”. The Washington Post reported that vaccinated people represented 58% of the COVID-19 deaths in August 2022. Indeed, the CDC registered a total of 6,512 COVID-19 deaths that month, of which 2,719 occurred among unvaccinated people and 3,793 occurred among vaccinated people.

However, these data aren’t corrected for the many confounding factors that we detailed earlier. Mainly, they aren’t corrected for the age and the size of the vaccinated and unvaccinated population. In fact, the CDC reported that vaccinated people are less likely to die from COVID-19 in August 2022, once these confounding factors are taken into account. And the Washington Post article actually clarified that “vaccinated groups are at a lower risk of dying from a covid-19 infection than the unvaccinated when the data is adjusted for age”. But this detail is notably absent from many social media posts.

Therefore, contrary to the impression that certain social media posts gave, the Washington Post article actually didn’t contradict the CDC.

Assessing vaccine effectiveness

Instead of directly comparing the absolute number of COVID-19 deaths between vaccinated and unvaccinated groups, which would be affected by several biases as we explained, scientists usually calculate the vaccine effectiveness (VE), which measures the level of protection from a given outcome, such as death, compared to unvaccinated individuals. A VE of 60% indicates a 60% reduction in the risk of COVID-19 death in vaccinated individuals, compared with unvaccinated individuals.

There are many methods to calculate VE depending on the availability of data and the need to adjust for confounding factors. A simple method uses the IRR presented in figure 3. Other techniques use a regression model^[9] to estimate the effect of vaccination on the likelihood of dying from a disease.

The scientific literature reports that COVID-19 vaccines are highly effective against death, although the figure varies depending on the virus variant and the amount of time that has passed after vaccination^[10-13]. The report by the U.K. Health Security Agency cited earlier indicated a VE from 59% to 95% against death from Omicron variant infection and 90% against the Alpha and Delta variants.

Conclusion

Comparing COVID-19 mortality between vaccinated and unvaccinated populations provides us with important information to guide public health decisions. However, a crude comparison of death counts can lead to ill-founded conclusions, as many confounding factors may bias the analysis. Among them, the level of vaccine coverage and the age distribution of the vaccinated and unvaccinated groups are the most common confounders but are easy to account for.

Flawed analyses disregarding these confounding factors have been used to build the false narrative that COVID-19 vaccines are ineffective at protecting against death from COVID-19. These analyses commonly compared crude death counts without factoring in the extent of vaccine coverage in the population. In contrast, more rigorous scientific analyses that account for the effect of confounding factors showed that COVID-19 vaccines significantly reduce the odds of vaccinated people dying from the disease compared to unvaccinated individuals.

REFERENCES

• 1 – Polack et al. (2020) Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine. The New England Journal of Medicine.
• 2 – Baden et al. (2021) Efficacy and Safety of the mRNA-1273 SARS-CoV-2 Vaccine. The New England Journal of Medicine
• 3 – Si et al. (2021) Investigating the Links Between Vaccination Against COVID-19 and Public Attitudes Toward Protective Countermeasures: Implications for Public Health. Frontiers in Public Health.
• 4- Andersson et al. (2021) Anticipation of COVID-19 vaccines reduces willingness to socially distance. Journal of Health Economics.
• 5 – Makhoul et al. (2020) Epidemiological Impact of SARS-CoV-2 Vaccination: Mathematical Modeling Analysis. Vaccines.
• 6 – Usherwood et al. (2021) A model and predictions for COVID-19 considering population behavior and vaccination. Scientific Reports.
• 7 – Ozasa & Fukushima (2019) Commentary: Test-Negative Design Reduces Confounding by Healthcare-Seeking Attitude in Case-Control Studies. Journal of Epidemiology.
• 8 – De Serres et al. (2013) The test-negative design: validity, accuracy and precision of vaccine efficacy estimates compared to the gold standard of randomized placebo-controlled clinical trials. Eurosurveillance.
• 9 – Venmani (2019) Comparison of regression models on estimation of vaccine efficacy in anti-leprosy vaccination trial-a large prospective vaccination trial. AIP Conference Proceedings.
• 10 – Zheng et al. (2022) Real-world effectiveness of COVID-19 vaccines: a literature review and meta-analysis. International Journal of Infectious Diseases.
• 11 – Andrews et al. (2022) Effectiveness of COVID-19 booster vaccines against COVID-19-related symptoms, hospitalization and death in England. Nature medicine.
• 12 – Ioannou et al. (2021) COVID-19 Vaccination Effectiveness Against Infection or Death in a National U.S. Health Care System. Annals of Internal Medicine.
• 13 – Sheikh et al. (2021) BNT162b2 and ChAdOx1 nCoV-19 Vaccine Effectiveness against Death from the Delta Variant. The New England Journal of Medicine.