Study claiming that ivermectin reduces COVID-19 mortality by 92% has important methodological problems

In early September 2022, many social media posts and articles circulated claiming that a study showed ivermectin reduced COVID-19 mortality by 92%. One example is this article by The Blaze, which received more than 20,000 engagements on social media, including more than 11,000 shares on Facebook, according to social media analytics tool CrowdTangle. The claim is based on a study by Kerr et al., published in the journal Cureus^[1].

This claim sometimes went hand in hand with another claim that the U.S. National Institutes of Health (NIH) now considers ivermectin to be a COVID-19 therapy, as seen in this Facebook post by music video director Robby Starbuck and this tweet by mixed martial artist Jake Shields. Starbuck’s post referenced the NIH COVID-19 Treatment Guidelines webpage, which lists the drugs currently being evaluated for their effectiveness against COVID-19.

But as we will explain below, the first claim is unsubstantiated and misleading, while the second is inaccurate.

Study by Kerr et al. used methods that produced bias and led to significant limitations in the reliability of its conclusions

In the study, the authors evaluated whether regular ivermectin use influenced the risk of COVID-19 mortality, using data from a clinical database for a COVID-19 prevention program in the Brazilian city of Itajaí. The program ran from July to December 2020. The authors compared COVID-19 mortality between regular ivermectin users and people who used ivermectin irregularly or not at all. Based on their analysis, they concluded that the “[m]ortality rate was 92% lower in regular users than non-users”.

However, scientists who examined the study found that it wasn’t performed in a way that would enable reliable conclusions to be made about ivermectin’s effect on COVID-19 mortality risk.

In fact, the authors published a study on the same dataset in January 2022, claiming that ivermectin, when used in a preventative manner, reduced COVID-19 mortality and hospitalization by half^[2]. That study was similarly panned for its methodological flaws. You can read comments by scientists about that study in this Health Feedback review.

One critical problem with the authors’ methods was pointed out by Greg Tucker-Kellogg, a professor of biology at the National University of Singapore, courtesy of physician Kyle Sheldrick. The authors had defined regular users of ivermectin as those who took “180 mg or more of ivermectin” in total throughout the program, and irregular users were those who took “up to 60 mg, in total, throughout the program”.

But one caveat is that if a participant in the program developed COVID-19, they were recommended not to use ivermectin. This detail about the Itajaí COVID-19 prevention program was stated in the authors’ first study that used the same dataset^[2]:

“In cases where a participating citizen of Itajaí became ill with COVID-19, they were recommended not to use ivermectin or any other medication in early outpatient treatment.”

This detail is absent from the recent study.

Because of the way that the authors defined regular users (those who took a total of 180 mg or more of ivermectin by the end of the program), people who stopped taking ivermectin after developing COVID-19 during the study were very likely excluded from the group of regular users—even if they had been regularly taking ivermectin up until that point—and placed in the group of irregular users or the group of intermediate users (people who took between 60 to 180 mg of ivermectin by the end of the program) instead.

There are two issues with this. Firstly, excluding those with a COVID-19 diagnosis from the group of regular users—even if their ivermectin use up until that point had been regular—means that the group of regular users is overrepresented by those who didn’t catch COVID-19. As such, COVID-19 patients are overrepresented among ivermectin irregular and intermediate users. Indeed, the authors reported that of the 7,228 people who developed COVID-19 during the program, only 283 (3.9%) were considered as regular users.

This introduces a clear bias when calculating the risk of COVID-19 mortality based on ivermectin use, because only people who don’t develop COVID-19 at all are more likely to be included among regular users. And if someone doesn’t develop COVID-19, then they aren’t at risk of dying from COVID-19 in the first place. This bias would make it seem as if people who used ivermectin regularly are less likely to die from COVID-19 compared to those who took ivermectin irregularly or didn’t take it at all.

Secondly, based on the definitions of regular use and inclusion criteria set by the authors, those who took intermediate doses of ivermectin appear to have been excluded entirely from the study: “Of the 113,844 participants, 8,325 (7.3%) subjects used ivermectin regularly and 33,971 (29.8%) used ivermectin irregularly. In total, 88,012 subjects were included in the present analysis. The 71,548 (62.8%) remaining participants used intermediate doses between 60 mg and 180 mg and were not included in this analysis.” [emphasis added]

And of the 7,228 participants who were infected with COVID-19 during the study, 32.8% used intermediate doses of ivermectin, amounting to almost a third of COVID-19 patients who did use ivermectin.

Overall, the authors’ methods impose significant limitations that raise questions about what the authors’ conclusions would have been if they’d also considered the pattern of ivermectin use before infection and if the intermediate group’s outcomes had been included.

Epidemiologist Gideon Meyerowitz-Katz also pointed out that while the authors accounted for certain confounding factors like pre-existing health conditions, such as type 2 diabetes and hypertension, they didn’t account for the potential effect of other confounding factors, like income.

13/n For example, income is a huge confounder when it comes to analyses like these. Could income have potentially changed the results?

Well, based on the limitations section where the authors note that they had no data for private hospitals…probably, yes pic.twitter.com/IwWTM1KMuF

— Health Nerd (@GidMK) September 4, 2022

Confounding factors are variables that affect the outcome of an experiment, but aren’t the variables being studied in the experiment. For example, scientists may wish to study the causes of heart disease. They may observe that red meat consumption is associated with a higher risk of heart disease. But it is possible that people who tend to eat more red meat might also be more likely to smoke cigarettes. In such a case, one possible explanation for the observation is that cigarette smoking, rather than red meat consumption, is the true influence behind that risk. This makes cigarette smoking a confounding factor.

This illustrates how failing to account for such confounding factors can lead scientists to draw erroneous conclusions about the cause of an effect that they observe in the study.

Meyerowitz-Katz also noted several inaccuracies in the title of the study. For instance, the study claimed to be a prospective, observational study, but is in fact a retrospective study.

6/n As the methods clearly states, the study design and ethics approval came AFTER the public health program occurred. This is more correctly termed a retrospective observational study of routinely recorded medical records pic.twitter.com/bdqjPp4pBG

— Health Nerd (@GidMK) September 4, 2022

Large, randomized controlled trials have already found that ivermectin was of no benefit to COVID-19 patients. In June 2022, a Cochrane Systematic Review that analyzed these trials reported^[3]:

“For outpatients, there is currently low- to high-certainty evidence that ivermectin has no beneficial effect for people with COVID‐19. Based on the very low‐certainty evidence for inpatients, we are still uncertain whether ivermectin prevents death or clinical worsening or increases serious adverse events, while there is low‐certainty evidence that it has no beneficial effect regarding clinical improvement, viral clearance and adverse events.”

The TOGETHER trial, which was performed across 12 public health clinics in Brazil, compared 679 COVID-19 patients assigned to receive ivermectin [(400 microgram per kilogram of body weight) once daily for 3 days] with 679 others who received a placebo. It reported no difference in the study’s primary outcomes, defined as hospitalization due to COVID-19 or emergency room visit due to worsening of COVID-19 within 28 days of treatment assignment^[4]:

“Treatment with ivermectin did not result in a lower incidence of medical admission to a hospital due to progression of Covid-19 or of prolonged emergency department observation among outpatients with an early diagnosis of Covid-19.”

The I-TECH trial, which was conducted in Malaysia, studied patients who were administered the standard of care together with oral doses of ivermectin for five days during the first week of illness. The researchers detected no difference in the likelihood of developing severe COVID-19 between patients given ivermectin and patients given standard of care only^[5].

In summary, the available evidence so far, including clinical trials in humans, hasn’t indicated that the drug is effective against the disease. And the conclusions of the study by Kerr et al., owing to its methodological flaws, are unreliable and insufficient to overturn the current body of evidence indicating that ivermectin is ineffective against COVID-19.

The NIH recommends against treating COVID-19 with ivermectin; this recommendation hasn’t changed

Some social media posts claimed that the U.S. National Institutes of Health (NIH) recently added ivermectin as a COVID-19 treatment. For example, this Facebook post by Starbuck references the NIH COVID-19 Treatment Guidelines webpage, which lists the drugs currently being evaluated for their effectiveness against COVID-19.

However, ivermectin was listed on this page as early as June 2021 and isn’t a recent addition, as we can see if we compare archives of the webpage (see below).

How the NIH COVID-19 Treatment Guidelines page on antiviral therapy appeared at different points in time. Top: Archive from June 2021. Bottom: Archive from September 2022.

In fact, if one simply clicked on “Ivermectin” on the NIH page, the NIH makes it clear that they recommend against using ivermectin to treat COVID-19, except in a clinical trial.

 

REFERENCES

• 1 – Kerr et al. (2022) Regular Use of Ivermectin as Prophylaxis for COVID-19 Led Up to a 92% Reduction in COVID-19 Mortality Rate in a Dose-Response Manner: Results of a Prospective Observational Study of a Strictly Controlled Population of 88,012 Subjects. Cureus.
• 2 – Kerr et al. (2022) Ivermectin Prophylaxis Used for COVID-19: A Citywide, Prospective, Observational Study of 223,128 Subjects Using Propensity Score Matching. Cureus.
• 3 – Popp et al. (2022) Ivermectin for preventing and treating COVID‐19. Cochrane Database of Systematic Reviews.
• 4 – Reis et al. (2022) Effect of Early Treatment with Ivermectin among Patients with Covid-19. New England Journal of Medicine.
• 5 – Lim et al. (2022) Efficacy of Ivermectin Treatment on Disease Progression Among Adults With Mild to Moderate COVID-19 and Comorbidities: The I-TECH Randomized Clinical Trial. JAMA Internal Medicine.