Country-to-country comparison doesn’t show that COVID-19 vaccines are ineffective, contrary to Facebook post by journalist Sharyl Attkisson

Country-to-country comparisons have been used to push the narrative that COVID-19 vaccines are ineffective, even though clinical trials and post-marketing surveillance proved the opposite^[1-3].

Science Feedback previously covered such claims and explained why directly comparing data from different countries can give rise to ill-founded conclusions.

A Facebook post by TV host Sharyl Attkisson published on 1 April 2024 is yet another example of such a claim. In the post, Attkisson claimed that Iceland and Portugal “had over 75% of their population fully vaccinated but had more Covid-19 cases per 1 million people than countries such as Vietnam and South Africa that had around 10% of their population fully vaccinated.” Attkisson previously propagated misinformation about vaccines and COVID-19, as documented by Science Feedback.

This comparison strongly implied that COVID-19 vaccines are ineffective. However, Attkisson’s claim is inaccurate. Furthermore, her underlying reasoning is flawed. As we explain below, there are important differences between these countries that make direct comparisons like Attkisson’s meaningless.

The claim’s figures don’t represent the latest vaccine coverage of Vietnam

To begin with, Attkisson’s figures for vaccine coverage in the four countries cited don’t represent the latest data available. According to the data aggregator Our World in Data, which cites data from the World Health Organization and national health departments, COVID-19 vaccine coverage was 87% in Vietnam as of 30 June 2023. This figure is the latest data available for this country.

COVID-19 vaccine coverage was 77% in Iceland as of 29 March 2022 and 86% in Portugal as of 29 March 2023. It was lower in South Africa, at 35% as of 24 September 2023 (Figure 1).

Figure 1 – Share of fully vaccinated individuals in Vietnam, Portugal, Iceland, and South Africa. Source: Our World in Data.

All these figures predate Attkisson’s post, which was made on 1 April 2024. Thus, the vaccine coverage in South Africa and Vietnam is now clearly higher than the proportions cited in Attkisson’s post. In particular, vaccine coverage in Vietnam is comparable with that of Iceland and Portugal.

By examining the data in the past, we noticed that the vaccine coverage in South Africa and Vietnam was only concomitantly “around 10%” in September 2021. Although this could be consistent with Attkisson’s claim, the number of COVID-19 cases at the time isn’t (Figure 2). The number of COVID-19 cases in September 2021 was higher in South Africa than in Iceland, contrary to the claim.

Figure 2 – Number of COVID-19 cases per million inhabitants of each country. Source: Our World in Data.

These inconsistencies alone are enough to invalidate Attkisson’s implication that COVID-19 vaccines don’t work, since this implication hinges on Vietnam and South Africa having much lower vaccine coverage and COVID-19 cases than Iceland and Portugal.

Differences in age structure are a confounding factor that may lead to bias

Apart from the numerical inconsistencies in Attkisson’s post, the logic underlying her implication is flawed. This is because the countries she referenced differ in aspects other than vaccine coverage, which impact the number of reported COVID-19 cases. These are known as confounding factors.

Confounding factors are variables that affect the outcome of an experiment, but aren’t the variables being studied in the experiment. If these other factors aren’t taken into account, it’s not possible to reliably establish whether an observed difference in COVID-19 cases between two countries is due to a difference in vaccine coverage, some of these other factors, or a combination of both.

One important confounding factor is the different population age distribution between countries. Figure 3 shows that the average number of confirmed cases in 2022 is proportional to the share of individuals above 65 in the population.

This is relevant to Attkisson’s comparison because Iceland, Portugal, Vietnam, and South Africa have different population age structures: the share of people above 65 is higher in Iceland and Portugal than in Vietnam and South Africa (Figure 3).

Figure 3 – Correlation between the number of COVID-19 cases and individuals older than 65 in each country, expressed as a percentage of the total population. COVID-19 cases are expressed as the average of all the reports of the cumulative number of confirmed cases available for 2022 for the 180 countries in the dataset^#. The colored dots represent the four countries used in Attkisson’s claim. Orange: South Africa, red: Vietnam, blue: Iceland, green: Portugal. Source: Our World in Data.

Therefore, one can expect to observe more COVID-19 cases in countries with older populations because of the correlation between age and number of COVID-19 cases. If this effect isn’t accounted for, any observed difference in the number of COVID-19 cases between countries could be due to the differences in their population age distribution, rather than their vaccine coverage, like Attkisson claimed.

Differences in the age distribution of populations also impact the detection of COVID-19 cases. We know that COVID-19 disproportionately affects older people, who are more at risk of developing symptomatic and severe COVID-19. For that reason, older people were also among the first to get vaccinated when vaccines became available. By contrast, younger people are more often asymptomatic^[4,5].

Without a systematic screening system in place, asymptomatic cases are more likely to go undiagnosed and, therefore, unrecorded in the data. Thus, countries with older populations are more likely to exhibit a higher rate of infection as well as a higher vaccine coverage.

Differences in COVID-19 detection and reporting may be another source of bias

Accurate reporting of COVID-19 cases requires a healthcare system able to provide large-scale population testing and timely recording of positive tests. If a country isn’t able to do so, the official number of COVID-19 cases will be lower than the actual number of cases. Therefore, if one country suffers from significantly more underreporting compared to others, this will bias the country-to-country comparison of COVID-19 cases.

According to an article by the New York Times, the lack of COVID-19 testing underestimated the real extent of COVID-19 infections in Africa^[6]. While the reported number of cases was low, seroprevalence studies indicated that 65% of the African population had in fact been infected by the third quarter of 2021. Epidemiological modeling studies also concluded that African countries suffered from an underreporting of COVID-19 cases^[7,8].

By contrast, underreporting is likely to be less of an issue in higher-income countries like Portugal and Iceland. Therefore, it’s plausible that the lower number of cases officially reported in South Africa is partly a consequence of underreporting and not because it has lower vaccine coverage. But Attkisson’s post takes all official figures at face value without accounting for this limitation.

Differences in adherence to COVID-19 measures also affect the number of COVID-19 cases

Many countries issued guidelines and enforced measures to limit the spread of COVID-19. However, the reactiveness and the nature of those measures, and how well people adhered to those measures, differed greatly between countries^[9,10]. Therefore, it’s possible that differences in COVID-19 cases between countries can be explained in part by differences in how well each population adhered to COVID-19 measures. In other words, adherence to public health measures is yet another important confounding factor that should be considered before comparing countries.

With that in mind, it’s important to note that Vietnam was able to curb the first wave of COVID-19 thanks to its rapid response and its population’s high degree of adherence to COVID-19 measures^[11,12].

Therefore, it’s plausible that the lower number of COVID-19 cases in Vietnam compared to some other countries can be explained in part by better adherence with COVID-19 guidelines. Yet, Attkisson’s analysis failed to take this factor into account.

Conclusion

Country-to-country comparisons are relatively easy to make because country-level statistics, such as the percentage of vaccinated people or the number of people with COVID-19, are readily available.

However, analyses using such data come with caveats. Specifically, one must pay attention to the existence of confounding factors that also affect the outcome of interest—in this case, the number of COVID-19 cases.

Indeed, the age of the population, compliance with physical distancing and hygiene guidelines, and COVID-19 reporting efficiency are all parameters that differ from country to country and impact how many people get the disease. Analyses, like Attkisson’s, that don’t consider these factors are thus flawed and don’t provide any insight into the COVID-19 vaccines’ effectiveness.

By contrast, clinical trials and post-marketing surveillance all show that COVID-19 vaccines are safe and effective at preventing severe disease.

NOTE

#: The dataset from Our World in Data contains periodical reports of the cumulative number of COVID-19 cases since the beginning of the pandemic, for each country. However, the frequency of those reports differs between countries and between years. We decided to average all the 2022 reports of that cumulative number instead of picking up one single time point to avoid the risk of cherry-picking. We chose to use data from 2022 because this was one year after vaccination rollout began and because data for Iceland, which is relevant to the claim, isn’t available for 2023.

REFERENCES

• 1 – Ssentongo et al. (2022) SARS-CoV-2 vaccine effectiveness against infection, symptomatic and severe COVID-19: a systematic review and meta-analysis. BMC Infectious Disease.
• 2 – Wang et al. (2022) Real-Word Effectiveness of Global COVID-19 Vaccines Against SARS-CoV-2 Variants: A Systematic Review and Meta-Analysis. Frontiers in Medicine.
• 3 – Yang et al. (2023) Efficacy of SARS-CoV-2 vaccines and the dose–response relationship with three major antibodies: a systematic review and meta-analysis of randomised controlled trials. The Lancet Microbe.
• 4 – Karron et al. (2022) Assessment of Clinical and Virological Characteristics of SARS-CoV-2 Infection Among Children Aged 0 to 4 Years and Their Household Members. JAMA Network Opens.
• 5 – Wang et al. (2023) Asymptomatic SARS-CoV-2 Infection by Age: A Global Systematic Review and Meta-analysis. The Pediatric Infectious Disease Journal.
• 6 – Lewis et al. (2022) SARS-CoV-2 infection in Africa: a systematic review and meta-analysis of standardised seroprevalence studies, from January 2020 to December 2021. BMJ Global Health.
• 7 – Thenon et al. (2022) COVID-19 in Africa: Underreporting, demographic effect, chaotic dynamics, and mitigation strategy impact. PLOS Neglected Tropical Diseases.
• 8 – Han et al. (2023) Estimation of epidemiological parameters and ascertainment rate from early transmission of COVID-19 across Africa. Royal Society Open Science.
• 9 – Lin et al. (2021) A multi-national test on self-reported compliance with COVID-19 public health measures: The role of individual age and gender demographics and countries’ developmental status. Social Science & Medicine.
• 10 – Georgieva et al. (2021) Perceived Effectiveness, Restrictiveness, and Compliance with Containment Measures against the Covid-19 Pandemic: An International Comparative Study in 11 Countries. International Journal of Environmental Research and Public Health.
• 11 – Vo et al. (2020) Adherence to Social Distancing Measures for Controlling COVID-19 Pandemic: Successful Lesson From Vietnam. Frontiers in Public Health.
• 12 – Nguyen et al. (2020) Preventive behavior of Vietnamese people in response to the COVID-19 pandemic. Plos One.