Significant methodological flaws in a 2020 study claiming to show unvaccinated children are healthier

A study published on 28 May 2020^[1] has been used to support a claim shared in articles and social media posts on Facebook and Instagram that unvaccinated children are healthier than vaccinated children. As is typical for this type of claim, the posts have most commonly been shared by Facebook groups that oppose vaccines, but also by groups that promote conspiracy theories.

This claim is not new. A study by Mawson et al. in 2017 was used similarly by vaccine skeptics. Snopes found that study to be fraught with methodological problems and flawed statistical analyses which invalidated its conclusions.

The 2020 study examined the medical records of patients from three pediatric practices as a “convenience sample”, selecting records with diagnoses of developmental delay, asthma, ear infection, and gastrointestinal disorder. The study authors did not clearly describe how these pediatric practices were selected. They then compared the number of vaccinated children who had received any of the four diagnoses to the number of unvaccinated children, and concluded that vaccination is associated with a higher incidence of developmental delays, asthma, and ear infections. As a control, a diagnosis of head injury was used since it is a health outcome unlikely to be related to vaccination.

Scientists who evaluated the study told Health Feedback that it contains numerous methodological flaws, one of which is the non-representative sample population. Karina Top, an associate professor of pediatrics at Dalhousie University, pointed out that the proportion of unvaccinated children in the study was much higher than that in the general population. According to a 2019 CDC report on vaccine coverage, only 1.3% of U.S. children had received no vaccinations at two years of age, yet “30% of children in their sample of three pediatric practices had received no vaccines,” she said.

This raises questions about the type of pediatric practices included in the study. If the physicians at these practices were unsupportive of vaccination or more willing to provide medical exemptions for vaccination, these clinics would have drawn families who are more vaccine-hesitant or who object to vaccines for various reasons, she explained.

The non-representative sample is likely to have arisen due to the use of convenience sampling in the study. David Gorski, a professor of surgery at Wayne State University and editor of the website Science-Based Medicine, explained in his blog post that while the study’s method of convenience sampling makes it easy to assemble a study population, this method suffers from several problems:

“[T]he main one being that [convenience samples] are rarely representative of the general population and therefore cannot be generalized. Others include bias and over- or underrepresentation of the population. Basically, no matter how you analyze a convenience sample, you can’t generalize it to the larger population.”

Apart from the non-representative sample population, Wagner pointed out that “A large problem with this study is that the researchers did not control for differences between the groups of unvaccinated and vaccinated children.”

Controlling for differences between vaccinated and unvaccinated children is important, as vaccination status itself is associated with other factors that can influence health outcomes but do not result from vaccination itself. For example, vaccinated children are more likely to see a doctor when unwell compared to unvaccinated children for various reasons, such as socioeconomic status, accessibility to healthcare services, and possibly greater trust in healthcare professionals^[2,3].

As a result, vaccinated children are much more likely to be diagnosed with medical conditions, but this does not necessarily mean that they are more likely to develop such conditions in the first place. Nina Masters, a PhD student in epidemiology at the University of Michigan, points out that the authors were aware of this bias, as they stated that “A single significant relationship was seen for the head injury control diagnosis at the 18-month vaccination cut-off, which may be indicative of differences in healthcare-seeking behavior among families of vaccinated versus unvaccinated children.” But the authors did not follow up on this by informing the reader of its significance or how it might affect their conclusions.

Notably, the first author of the 2020 study is Brian Hooker, a chemical engineer who previously published a now-retracted study purportedly showing higher rates of autism in African-American boys who had been vaccinated. The study had used “fraudulent methods and failed to disclose conflicts of interest,” said Wagner. Health Feedback also covered the retracted study in an earlier review. The second author of the 2020 study is Neil Z. Miller, a journalist without any training in biology or medicine, who has published other questionable studies in the past.

By contrast, several well-designed studies examining differences in health and developmental outcomes between vaccinated and unvaccinated children have not detected adverse health outcomes in vaccinated children. A 2004 study in Pediatrics showed no association between vaccines and developmental delay^[4]. Another study found that children who had been vaccinated in the first year of life performed better on cognitive tests^[5]. Similarly, measles vaccination in developing countries, specifically Ethiopia, India, and Vietnam, was associated with better cognitive test scores^[6]. A 2011 study in Germany, which examined the incidence of allergies and infections among more than 13,000 individuals, did not find adverse health outcomes associated with vaccination^[7]. Another study in Germany, published in 2014, examined more than 1,300 individuals and found that vaccination was associated with a significantly lower incidence of asthma^[8]. A 2020 Cochrane Review of 138 studies showed no evidence supporting an association of MMR vaccination with asthma, bacterial or viral infections, cognitive delay, type 1 diabetes, dermatitis/eczema, and hay fever^[9]. At least 20 studies have shown that vaccines are not associated with autism^[4,10-29], as this Health Feedback review discussed.

Vaccines are safe and effective. The U.S. Institute of Medicine concluded in a 2013 review that the childhood immunization schedule is safe^[12]. The Vaccine Education Center at the Children’s Hospital of Philadelphia has also summarized the scientific evidence showing that vaccines are not associated with a higher risk of asthma or allergies and neurodevelopmental problems like attention deficit/hyperactivity disorder. The American Academy of Pediatricians has also compiled a list of studies relevant to vaccine safety here.

Vaccines are safe and effective. Unvaccinated children can get terrible diseases—an unvaccinated 6-year old boy in Oregon was diagnosed with tetanus after having uncontrollable muscle spasms and he was hospitalized for 8 weeks. The Hib vaccine protects against epiglottitis—the swelling of the throat which can cause infants to suffocate. The whooping cough vaccine protects against a disease where children can cough until they throw up and break their ribs.

A recent study examined the relationship between the number of vaccines administered and different health outcomes. A large problem with this study is that the researchers did not control for differences between the groups of unvaccinated and vaccinated children. We know vaccinated and unvaccinated children can come from different environments: living in rural (vs. urban) areas, wealth, proclivity to go to the doctor, etc. All these factors could differ between vaccinated and unvaccinated children and could explain differences in health outcomes. Additionally, this study includes children from handpicked medical practices and is not representative of the general population.

The first author’s previous publication was retracted for fraudulent methods and undisclosed conflicts of interest.

This analysis does not account for differential healthcare seeking between the vaccinated and unvaccinated. The authors do not evaluate whether there are different numbers of doctor’s visits between the two groups. For example, the unvaccinated group could be more likely to miss appointments with their doctor, which could lead to them receiving fewer vaccines and also having less opportunity for doctor’s visits in which to be diagnosed with various health conditions.

The authors even acknowledge that this bias may exist: “A single significant relationship was seen for the head injury control diagnosis at the 18-month vaccination cut-off, which may be indicative of differences in healthcare-seeking behavior among families of vaccinated versus unvaccinated children.” Yet they do not present any information that would enable the reader to better understand the role and scale of this bias.

Diagnosis with many developmental delays may occur in the 3-5 year range, but growing evidence has shown that the factors that lead to these diagnoses occur early in life and during prenatal development—long before any vaccination.

First, we know that the large majority of parents do choose to follow vaccine recommendations and at age two only 1.3% of U.S. children had received NO vaccinations. The finding that 30% of children in their sample of three pediatric practices had received no vaccines raises a red flag about the type of practice/physician and patients in their practice. Were the physicians not supportive of vaccines or willing to give medical exemptions and therefore attracted families that were more hesitant around vaccines or who had objections to vaccines for religious, cultural or other reasons? No details are provided regarding how they chose the practices, their location, or type of insurance they accepted (e.g. private, Medicaid).

Because the large majority of children are vaccinated, we know that unvaccinated children are very different from vaccinated children in ways that may also alter their likelihood of being diagnosed with childhood conditions such as asthma, ear infections, and development delay. For example, children from large families with low socioeconomic status may have difficulty getting to vaccination appointments, but those same challenges may make it difficult to get to a physician appointment for a new health problem. The analysis did not take into account demographic or other factors that might influence both a child’s chance of getting vaccinated and their chance of getting diagnosed with any of those conditions (e.g. insurance status, parent age, education, race/ethnicity, presence of other children).

Finally, both authors are well known for promoting unscientific claims about potential harms of vaccines, including the myth of an association between vaccines and autism for which the lead author has had two of his publications retracted by journals, suggesting an inherent bias in their approach.

REFERENCES

• 1 – Hooker and Miller. (2020) Analysis of health outcomes in vaccinated and unvaccinated children: Developmental delays, asthma, ear infections and gastrointestinal disorders. SAGE Open Medicine.
• 2 – Thomson et al. (2016) The 5As: A practical taxonomy for the determinants of vaccine uptake. Vaccine.
• 3 – Salmon et al. (2005) Factors Associated With Refusal of Childhood Vaccines Among Parents of School-Aged Children: A Case-Control Study. Archives of Pediatrics and Adolescent Medicine.
• 4 – Andrews et al. (2004) Thimerosal Exposure in Infants and Developmental Disorders: A Retrospective Cohort Study in the United Kingdom Does Not Support a Causal Association. Pediatrics.
• 5 – Smith and Woods. (2010) On-time Vaccine Receipt in the First Year Does Not Adversely Affect Neuropsychological Outcomes. Pediatrics.
• 6 – Nandi et al. (2019) Anthropometric, cognitive, and schooling benefits of measles vaccination: Longitudinal cohort analysis in Ethiopia, India, and Vietnam. Vaccine.
• 7 – Schmitz et al. (2011) Vaccination Status and Health in Children and Adolescents: Findings of the German Health Interview and Examination Survey for Children and Adolescents (KiGGS). Deutsches Ärzteblatt International.
• 8 – Grabenhenrich et al. (2014) Early-life Determinants of Asthma From Birth to Age 20 Years: A German Birth Cohort Study. Journal of Allergy and Clinical Immunology.
• 9 – Di Pietrantonj et al. (2020) Vaccines for measles, mumps, rubella, and varicella in children. Cochrane Database of Systematic Reviews.
• 10 – Madsen et al. (2002) A Population-Based Study of Measles, Mumps, and Rubella Vaccination and Autism. New England Journal of Medicine.
• 11 – Institute of Medicine. (2013). Adverse Effects of Vaccines: Evidence and Causality. Retrieved from https://doi.org/10.17226/13164
• 12 – Institute of Medicine. (2013). Childhood Immunization Schedule and Safety: Stakeholder Concerns, Scientific Evidence, and Future Studies. Retrieved from https://doi.org/10.17226/13563
• 13 – Institute of Medicine. (2004). Immunization Safety Review: Vaccines and Autism. Retrieved from https://doi.org/10.17226/10997
• 14 – Fombonne et al. (2006). Pervasive developmental disorders in Montreal, Quebec, Canada: prevalence and links with immunizations. Pediatrics.
• 15 – Taylor et al. (2014). Vaccines are not associated with autism: an evidence-based meta-analysis of case-control and cohort studies. Vaccine.
• 16 – Ball et al. (2001) An assessment of thimerosal use in childhood vaccines. Pediatrics.
• 17 – Hviid et al. (2003) Association Between Thimerosal-Containing Vaccine and Autism. JAMA.
• 18 – Madsen et al. (2003) Thimerosal and the occurrence of autism: negative ecological evidence from Danish population-based data. Pediatrics.
• 19 – Stehr-Green et al. (2003) Autism and thimerosal-containing vaccines: lack of consistent evidence for an association. American Journal of Preventive Medicine.
• 20 – Verstraeten et al. (2003) Safety of thimerosal-containing vaccines: a two-phased study of computerized health maintenance organization databases. Pediatrics.
• 21 – Thompson et al. (2007) Early Thimerosal Exposure and Neuropsychological Outcomes at 7 to 10 Years. New England Journal of Medicine.
• 22 – McMahon et al. (2008) Inactivated influenza vaccine (IIV) in children <2 years of age: examination of selected adverse events reported to the Vaccine Adverse Event Reporting System (VAERS) after thimerosal-free or thimerosal-containing vaccine. Vaccine.
• 23 – Schechter and Grether. (2008) Continuing increases in autism reported to California’s developmental services system: mercury in retrograde. Archives of General Psychiatry.
• 24 – DeStefano F. (2009) Thimerosal-containing vaccines: evidence versus public apprehension. Expert Opinion on Drug Safety.
• 25 – Tozzi et al. (2009) Neuropsychological performance 10 years after immunization in infancy with thimerosal-containing vaccines. Pediatrics.
• 26 – Price et al. (2010) Prenatal and Infant Exposure to Thimerosal From Vaccines and Immunoglobulins and Risk of Autism. Pediatrics.
• 27 – Barile et al. (2012) Thimerosal exposure in early life and neuropsychological outcomes 7-10 years later. Journal of Pediatric Psychology.
• 28 – DeStefano et al. (2013). Increasing Exposure to Antibody-Stimulating Proteins and Polysaccharides in Vaccines Is Not Associated with Risk of Autism. Journal of Pediatrics.
• 29 – Uno et al. (2012). The combined measles, mumps, and rubella vaccines and the total number of vaccines are not associated with development of autism spectrum disorder: the first case-control study in Asia. Vaccine.