Analysis of adverse event variation between Pfizer COVID-19 vaccine batches doesn’t indicate safety problems, contrary to claim by John Campbell

On 5 July 2023, YouTuber and retired nurse instructor John Campbell published a YouTube video regarding findings published in a research letter by Schmeling et al. in the European Journal of Clinical Investigation^[1]. In the video, which received more than 480,000 views, Campbell implied that COVID-19 vaccines were unsafe, citing the number of adverse events reported in the letter.

A day later, he published an interview with physician Vibeke Manniche, who co-authored the research letter. During the interview, Manniche claimed that their findings constituted a “safety signal”.

Regarded as an evidence-based source of information about COVID-19 in 2020, Campbell—who has accrued nearly three million followers on YouTube to date—has since repeatedly published YouTube videos containing false or misleading claims about COVID-19 and COVID-19 vaccines. A list of related reviews published by Health Feedback can be found here.

Manniche also previously spread COVID-19 misinformation that was debunked by Danish fact-checking group TjekDet. In March 2020, Manniche called COVID-19 pandemic measures hysterical and falsely claimed that COVID-19 wasn’t actually spreading, writing on several occasions that there was no coronavirus epidemic on the way. In February 2021, she baselessly asserted that the Danish population had already achieved herd immunity.

Neither Campbell nor Manniche’s claims are sufficiently supported by the published research letter, as we will explain below.

What the study did

The authors analyzed the number of suspected adverse events (SAEs) associated with more than 50 batches of the Pfizer-BioNTech COVID-19 vaccine, using publicly available data from the Danish Medicines Agency. The authors explained that this reporting system is analogous to the U.S. Vaccine Adverse Events Reporting System (VAERS).

The authors reported that there was a wide variation in the number of SAEs (not to be confused with “serious adverse events” which is also abbreviated in the same way) depending on the batch. They postulated that this variation could be due to differences in “vaccine manufacturing, storage, transportation, clinical handling and control aspects”.

The authors concluded that “the results suggest the existence of a batch-dependent safety signal for the BNT162b2 vaccine, and more studies are warranted to explore this preliminary observation and its consequences”.

But given that the reporting system bears the same limitations as VAERS, the authors acknowledged that “signals detected by these systems must be considered to be hypothesis-generating and generally cannot be used to establish causality”.

Reporting rate for vaccine batches can vary for reasons unrelated to vaccine safety

Campbell’s videos however, appeared to disregard the limitations stated by the authors. He claimed that “these high rates of potential side effects, one in 20 vaccines given” would have stopped him from getting the COVID-19 vaccine, thus implying that the numbers reported are unusual and show a safety problem.

However, the findings in the letter don’t provide the information needed for this implication. Campbell’s implied claim uses a very large number of post-vaccine adverse events as evidence that something is wrong with the vaccine.

This misleading tack has been employed numerous times in false claims about vaccine safety. It ignores the fact that if one monitors a very large population, there will be a baseline rate of illnesses and deaths that occur, even if this population is unvaccinated.

This STAT News article explained in detail:

“Every single day, people die unexpectedly. They have strokes and heart attacks and seizures. On an average day, 110 people in this country may develop Bell’s palsy, a temporary facial paralysis, and another 274 will develop Guillain-Barré syndrome, a form of paralysis that usually resolves over time. The trigger for these medical events often isn’t known. But when they happen shortly after someone gets a vaccine — especially a new one — well, conclusions will be drawn. […]

Heart attacks occur most commonly in the morning, yet we don’t blame breakfast for causing them. A heart attack on the morning after a Covid-19 vaccine, though? That might be another matter.”

The letter didn’t compare the rate of adverse events in vaccinated people with the baseline rates of such events. Therefore, it cannot be used to draw conclusions about whether vaccination raises the overall risk of illness and death, as Campbell did.

Manniche’s claim that the wide variation in SAEs necessarily points to a safety signal was refuted in this article by the Danish fact-checking group TjekDet, published in September 2022.

TjekDet interviewed Anton Pottegård, a pharmacoepidemiologist and professor at the University of Southern Denmark, who explained that such variation could be due to various factors that influence adverse event reporting rate, and not necessarily due to batch-related safety problems.

One possible reason is the improved understanding over time of which vaccine side effects are medically important. This could change the reporting rate over time, such that there would be more reports related to earlier batches compared to later batches.

Scientist Susan Oliver also remarked on this possibility in a YouTube video about Campbell’s claim, pointing to the Weber effect. The Weber effect refers to the phenomenon in which reports of adverse events related to a drug tend to peak around one to two years after the drug is approved, and decline afterwards. This is thought to be due to the perception that a drug’s side effect is better understood over time, leading people to be less motivated to report adverse events.

This is important to keep in mind, because during Manniche’s interview with Campbell, she stated that the dots for the blue trendline related to vaccines that were “given when we started to vaccinate which was in […] January, February, March 2021”. This information wasn’t available in the research letter, but based on this statement, we can infer that the green and yellow trendlines correspond to vaccine batches given later. Therefore, the change in adverse event reports reported in the letter is consistent with a potential Weber effect.

Pottegård highlighted that the Danish Medicines Agency had initially received many reports of mild side effects. The authorities later announced that people didn’t need to report flu-like symptoms or redness at the injection site following COVID-19 vaccination, which are some of the most common side effects.

The reporting culture could also differ between vaccination sites, Pottegård said. For instance, professionals in healthcare institutions may be more inclined to report compared to staff at vaccination centers. Thus the reporting rate for one batch could significantly differ from another depending on where the batches were sent.

The Danish Medicines Agency told TjekDet that potential batch-related side effects are investigated by the pharmaceutical authorities.

Anders Hviid, the head of epidemiology research at the Statens Serum Institute (SSI) and a professor of pharmacoepidemiology at the University of Copenhagen, addressed a letter to the editor of the journal that published the letter by Schmeling et al., published on 15 July 2023.

In the letter, Hviid pointed out that the authors had incorrectly described the data they used:

“They report that they have received batch-specific information on adverse event reports from the Danish Medicines Agency and administered vaccines from Statens Serum Institut (SSI). This is not correct. The information they have obtained from SSI is on shipped vaccine vials by batch number.”

He elaborated further:

“From SSI, vaccines were shipped to regional distribution centres (with −90 degree celsius storage capacity). The dates of data retrieval for both sources of data used in the letter are 18 January 2022 (shipment volumes) and 11 January 2022 (adverse event reports). Thus, batches shipped late in 2021 or very early in 2022 will not have any or very few adverse event reports, because they have not been used or only a small proportion have been used. This is also clearly seen in the figure in the letter, where the so-called ‘yellow’ batches have zero or close to zero reports. Understanding this is critical for the interpretation of these results. This is a particularly serious issue as the crux of the letter hinges on the idea that different vaccine batches have drastically different safety profiles which is not supported at all when you understand that it is not administered vaccines as they claim.”

He also added:

“There are many other issues with respect to these data. For example, that the very first batches were small, administered during the introduction period where clinical practitioners were instructed to report everything including local reactions, fever, transient headache, etc. These cannot be compared to later only partially used batches, where the authorities urged that mild and transient events should no longer be reported. It is also well-known that in the early part of the vaccination deployment, those vaccinating were urged to attempt to get seven doses out of the vials instead of six due to supply concerns. Again, as the authors do not have data on administered doses but only on shipped vials, the comparisons are further skewed.”

Hviid concluded by stating:

“In summary, the research letter is presenting the study data inaccurately and has a number of reporting inconsistencies that should also be corrected and even so the study data cannot be used to provide any meaningful insights into batch variability with respect to safety.”

Campbell and Manniche’s claim that the regulatory authorities don’t pay attention to post-marketing surveillance data or ignore this data is belied by the fact that the Danish authorities halted the use of the AstraZeneca COVID-19 vaccine following reports of blood clotting disorders post-vaccination.

Indeed, regulatory agencies around the world, including the U.S. Centers for Disease Control and Prevention, publicly acknowledged the possibility of rare side effects following COVID-19 vaccination, such as myocarditis and blood clots. It was vaccine adverse event reports that enabled authorities to recognize these as rare side effects of COVID-19 vaccines.

Multiple studies have shown that it is COVID-19 itself that is more likely to cause heart problems and blood clotting problems, not COVID-19 vaccination^[2-5].

Others took the findings of the letter even further, such as the anti-vaccine organization Children’s Health Defense, claiming that Schmeling et al. showed some people received “placebos” while others received “highly dangerous shots”. The language of this claim evokes early claims that COVID-19 vaccination is human experimentation, even though this isn’t true, as the vaccines underwent clinical trials before receiving authorization.

Health Feedback reached out to Manniche. In an email to Health Feedback, Manniche defended the work and insisted that “It is not premature to conclude the variation as a safety signal.” She stated that “If the batches were the same—including product, transportation, storage and distribution to the patients/public—you would find a homogeneity of the side effects. That is obviously not the case. We found an obvious heterogeneity.” (This assumption would certainly hold true if the reporting rate of adverse events remained constant over time. However, as Health Feedback explained above, several factors likely altered reporting rates over time.)

She objected to Health Feedback’s reporting, stating that we had not “understood the study properly” and calling our conclusions “totally wrong” and “speculative”.

We also reached out to Campbell and will update this review if new information becomes available.

COVID-19 vaccines did initially reduce transmission, remain effective against severe disease and death

Manniche’s claim that “We know that [the COVID-19 vaccine] didn’t do anything to transmission” is incorrect but embodies a common anti-vaccine trope: if a vaccine is less than 100% effective, it means it doesn’t work at all. This expectation is unrealistic, as there is no medical intervention or preventative that is 100% effective.

But simply because a vaccine works imperfectly doesn’t mean it doesn’t work at all (nirvana fallacy). The claim ignores the fact that COVID-19 vaccines originally did reduce virus transmission substantially^[6,7], but this was later undermined by the emergence of more infectious SARS-CoV-2 variants like Omicron.

Even so, this doesn’t make vaccination pointless, as the vaccines still retain the ability to significantly reduce the risk of severe disease and death.

Conclusion

The research letter by Schmeling et al. drew correlations between the number of suspected adverse events and three different batches of COVID-19 vaccines. While there is a wide variation in the number of reports from batch to batch, the letter provided no evidence that this variation is due to problems with safety. Other factors, such as a change in reporting rate due to improved understanding of vaccine side effects, could also have played a role. The authors didn’t do the work needed to rule out these factors. Multiple studies have shown that COVID-19 vaccines are highly effective at reducing the risk of severe disease and death, and that getting COVID-19 is riskier than getting vaccinated.

UPDATE (17 July 2023):

We updated our review to include quotes from a letter to the editor by Anders Hviid, pointing out that Schmeling et al. had incorrectly interpreted their data, in the twenty-fifth to the thirty-third paragraphs. We included a link to a YouTube video by scientist Susan Oliver, which also discussed Campbell’s claim, in the twentieth paragraph. We also added additional information from Campbell’s interview with Manniche in the twenty-first paragraph, clarifying that the blue trendline corresponds to vaccine batches used early on during the COVID-19 vaccination campaign.

UPDATE (13 July 2023):

We updated our review to include Manniche’s response to our request for comment in the thirty-eighth and thirty-ninth paragraphs.

REFERENCES

• 1 – Schmeling et al. (2023) Batch-dependent safety of the BNT162b2 mRNA COVID-19 vaccine. European Journal of Clinical Investigation.
• 2 – Barda et al. (2021) Safety of the BNT162b2 mRNA Covid-19 Vaccine in a Nationwide Setting. New England Journal of Medicine.
• 3 – Klein et al. (2021) Surveillance for Adverse Events After COVID-19 mRNA Vaccination. JAMA Network.
• 4 – Xie et al. (2022) Long-term cardiovascular outcomes of COVID-19. Nature Medicine.
• 5 – Xu et al. (2021) COVID-19 Vaccination and Non–COVID-19 Mortality Risk — Seven Integrated Health Care Organizations, United States, December 14, 2020–July 31, 2021. Morbidity and Mortality Weekly Report.
• 6 – Harris et al. (2021) Effect of Vaccination on Household Transmission of SARS-CoV-2 in England. New England Journal of Medicine.
• 7 – Amit et al (2021) Early rate reductions of SARS-CoV-2 infection and COVID-19 in BNT162b2 vaccine recipients. The Lancet.