Claims that a Harvard study showed COVID-19 vaccines are ineffective misrepresent the authors’ conclusions, fail to account for the study’s limitations

This video published by The Jimmy Dore Show on 17 October 2021 claimed in its title that a Harvard study showed that COVID-19 surged among most vaccinated communities. The cited study, which was published in the European Journal of Epidemiology on 30 September 2021, analyzed the association between the proportion of fully vaccinated people in the population and the prevalence of infection in 68 countries and 2,947 U.S. counties^[1].

According to social media analytics tool CrowdTangle, the study alone received more than 20,000 interactions on Facebook and more than 6,000 on Instagram, Reddit, and Twitter. The video itself received more than 216,000 views on YouTube. In addition, the study received further engagement through outlets like The Blaze, The Defender, and the Foundation for Economic Education (FEE), and websites opposing vaccination.

Robert Malone and Didier Raoult also shared the study on Twitter. Malone is a scientist who has repeatedly spread misinformation about COVID-19 vaccines. Raoult was a prominent promoter of hydroxychloroquine as a COVID-19 treatment, despite the lack of scientific evidence supporting its effectiveness. Altogether, articles and posts citing the study received more than 100,000 interactions on social media platforms.

In the video, the comedian Jimmy Dore quoted parts of this study to suggest that COVID-19 vaccines are ineffective, and that vaccine mandates are therefore useless. As we explain below, this interpretation of the study isn’t correct, as the study has important limitations that make this interpretation unfounded.

Simply correlating vaccine coverage with prevalence of infection doesn’t account for other factors that also influence the spread of the disease

The study analyzed data from 68 countries and 2,947 U.S. counties to establish whether there was a correlation between COVID-19 vaccine coverage and the number of COVID-19 cases per million people in the previous seven days. The data was retrieved on 3 September 2021 from Our World in Data, a project co-run by Oxford University, which aggregates health data all over the world from official sources.

The authors reported no correlation between the proportion of fully vaccinated people in the population and the rate of new COVID-19 cases in the last seven days, either at the country level or at the U.S. county level.

In the video, Dore discussed the study’s observations of different countries, such as Israel, which had the highest rate of COVID-19 cases, although more than 60% of its population was fully vaccinated. He also mentioned the higher prevalence of COVID-19 cases in highly vaccinated countries such as Iceland and Portugal compared to countries such as Vietnam and South Africa, which had much lower vaccine coverage (about 10%). By highlighting the observation that highly vaccinated countries still have many COVID-19 cases, the video left some viewers with the impression that COVID-19 vaccines are ineffective at tackling the pandemic, as is evident from comments claiming that the vaccine “isn’t working” and that the video showed the “real world ineffectiveness of vaccines”.

However, it is incorrect to assume from these data that COVID-19 vaccines don’t protect people from the disease. This is because the authors conducted an ecological study, which analyzes data at the population level rather than at the individual level. While ecological studies are quick and easy to conduct, they are also particularly prone to bias and confounding effects from unrelated factors that can distort the results if they aren’t taken into account. This type of study has often been used to spread anti-vaccine claims, as David Gorski, a professor of surgery at Wayne State University and an editor at Science-Based Medicine, pointed out on his blog Respectful Insolence.

One specific type of confounder in these studies lies in the ecological fallacy, which results from incorrectly assuming that the observations made in the population apply to each of the individuals as well. The epidemiologist Gideon Meyerowitz-Katz explained in his blog that any conclusions based on this assumption are unreliable because “the average of a group isn’t always representative of the individuals”. This doesn’t mean that we can’t draw any conclusions from population-level data, but that the correlations are complex and researchers have to avoid arriving at erroneous conclusions.

The lack of correlation between vaccination and the prevalence of infections at the population level doesn’t imply that COVID-19 vaccines don’t protect against COVID-19. Even in countries with a high level of vaccine coverage, a significant proportion of the population remains unvaccinated. These people are also more likely to be infected, meaning that they are more likely to make up the COVID-19 cases in the country. Therefore, the general population’s vaccine coverage is likely not representative of the specific population that got COVID-19.

The claim commits the ecological fallacy because vaccine coverage, which is measured at the population level, cannot be used to make assumptions about the vaccination status of every individual in the country. The study’s findings are unable to tell us about the effectiveness of the vaccine, since it is based on correlating case numbers with a population’s vaccine coverage, as opposed to case numbers with individuals’ vaccination status.

Besides, other factors, including the relaxation of restrictions, the spread of the more contagious Delta variant, and waning vaccine immunity in those people who received the vaccine first could explain the difference in the number of COVID-19 cases per million people between countries, or the rise in cases in highly vaccinated populations such as Israel.

It is also important to note that the primary goal of COVID-19 vaccines is not to prevent infections completely but to minimize the risk of severe disease and death. Therefore, measuring the effectiveness of COVID-19 vaccines solely based on the number of COVID-19 cases provides an incomplete picture about the vaccines’ effectiveness. While Israel saw a substantial rise in the number of COVID-19 cases during the third COVID-19 wave in September 2021, the number of deaths was much lower in comparison to previous waves (Figure 1). The uncoupling of cases from deaths demonstrates that COVID-19 vaccines effectively prevented severe disease and death.

Figure 1. Daily number of confirmed COVID-19 cases (top) and deaths (bottom) in Israel between 2 March 2020 and 20 October 2021. Data are presented as the rolling 7-day average. Data and image from Our World in Data.

The study didn’t take into account other factors that likely influenced the course of the pandemic in different countries or regions

The study isn’t a full research article but it was published as “Correspondence”, often called “Letters”. These publications are short reports typically intended for fast publication, and therefore, contain certain limitations.

The most important limitation in this study is that the authors didn’t account for the many factors that might influence the course of the pandemic in each region besides the vaccination rate. Such factors include socioeconomic and geographic differences, population densities, the quality of healthcare, or the overall health status of the population. Some of them can also directly impact the quality and reliability of the data used to make the correlations, as Carl Bergstrom, a biologist at the University of Washington, pointed out in this Twitter thread.

In addition, each country began the rollout of vaccination campaigns at different times, with different COVID-19 vaccines, and used different combinations of public health measures to curb the spread of the virus. All these potential confounders make population-level, and particularly international, comparisons often misleading.

Other aspects of the study add further concerns to the methodology used. One of them is that, despite being an epidemiological study, none of the study’s authors are epidemiologists. The first author, Sankaran Venkata Subramanian, is a Doctor of Geography and Professor of Population Health and Geography at Harvard University. His co-author, Akhil Kumar, is listed as a student from a high school in Ontario, Canada.

In addition, the methodology and conclusions of the study aren’t adequately backed up by previous scientific literature. In a research article, references to previous work are essential because they demonstrate the foundation upon which the study rests. Any statement that the article doesn’t experimentally demonstrate needs to be supported by references to one or more previous peer-reviewed studies supporting it. However, most of the references listed in the publication aren’t scientific studies, but links to news articles and public databases.

Claims that COVID-19 vaccines are ineffective misrepresent the conclusions of the study

Subramanian explained to the fact-checking organization PolitiFact that articles and posts using his study to claim that COVID-19 vaccines were ineffective misinterpreted the conclusions from the study:

“Concluding from this analysis that vaccines are useless is misleading and inaccurate. Rather, the analysis supports vaccination as an important strategy for reducing infection and transmission, along with handwashing, mask-wearing, proper ventilation and physical distancing.”

Subramanian’s statement is in line with general recommendations from public health authorities. None of the protective measures against COVID-19 are 100% effective on their own, as this figure by virologist Ian Mackay illustrates. . For this reason, the U.S. Centers for Disease Control and Prevention (CDC) stresses the importance of combining different protective measures, such as vaccinating, mask-wearing, handwashing, and physical distancing. These different layers of protection minimize the risk of infection, as Health Feedback explained in earlier reviews.

Vaccination remains the best protective strategy against COVID-19

The fact that vaccination, isn’t 100% effective on its own doesn’t render it useless. Dismissing a solution because it is imperfect, even though no perfect solution exists, is known as the nirvana fallacy.

Because vaccines don’t prevent 100% of infections, a small proportion of vaccinated people will still get infected and develop COVID-19 as long as the virus circulates. However, vaccinated individuals are still less likely to get infected compared to unvaccinated people, even in the presence of the more contagious Delta variant^[2-4].

One study conducted in Los Angeles County, California, when the Delta variant was already dominant, found that fully vaccinated people were 4.9 times less likely to get infected and 29.2 times less likely to get hospitalized compared to unvaccinated people^[5].

COVID-19 vaccines remain highly effective against severe disease and death, even though their effectiveness against infection seems to decline over time and can vary depending on the virus variant. One study published in the New England Journal of Medicine evaluated the effectiveness of the Pfizer-BioNTech COVID-19 after six months. Although vaccine effectiveness against symptomatic disease dropped to 84%, the effectiveness against severe disease after six months was 96.7%, similar to the efficacy reported in clinical trials^[6].

Conclusion

In summary, claims that COVID-19 vaccines are ineffective based on a lack of correlation between vaccine coverage and the prevalence of infection in several countries and U.S. counties misinterpret the conclusions of the study. Furthermore, the study has important design and methodological limitations that don’t allow us to draw any conclusions about how much COVID-19 vaccines protect the individuals. No vaccine is 100% effective and a small proportion of vaccinated people will still get the disease and even die. However, the scientific evidence shows that COVID-19 vaccines reduce the risk of infection and are highly effective at preventing severe disease and death.

REFERENCES

• 1 – Subramanian and Kumar. (2021) Increases in COVID-19 are unrelated to levels of vaccination across 68 countries and 2947 counties in the United States. European Journal of Epidemiology.
• 2 – Rosenberg et al. (2021) New COVID-19 Cases and Hospitalizations Among Adults, by Vaccination Status — New York, May 3–July 25, 2021. Morbidity and Mortality Weekly Report.
• 3 – Lopez Bernal et al. (2021) Effectiveness of Covid-19 Vaccines against the B.1.617.2 (Delta) Variant. New England Journal of Medicine.
• 4 – Pouwels et al. (2021) Effect of Delta variant on viral burden and vaccine effectiveness against new SARS-CoV-2 infections in the UK. Nature Medicine.
• 5 – Griffin et al. (2021) SARS-CoV-2 Infections and Hospitalizations Among Persons Aged ≥16 Years, by Vaccination Status — Los Angeles County, California, May 1–July 25, 2021. Morbidity and Mortality Weekly Report.
• 6 – Thomas et al. (2021) Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine through 6 Months. New England Journal of Medicine.