Process – How Science Feedback works

Our Workflow at a glance:

1. Select an item for assessment, according to subject matter, relevance, and potential digital influence
2. Investigate selected item, with a focus on assessing (claimed) fact-based assertions and scientific reasoning
3. Write review explaining the current state of knowledge in science
4. Publicize our review

• Credibility ratings & verdicts
  • Article review ratings
  • Claim review verdicts

---

Science Feedback editors select claims or articles for review that are the most viral on social media and/or are published by sources with large readership. Either must contain potentially verifiable claims in the scientific realm.

We strive for our reviews to be representative of the spectrum of influential discussion in the media and on social media. We review articles and claims from a variety of sources, without a priori perspective and regardless of whether they insightfully report on, exaggerate or downplay the science on climate and health issues.

Suggest an item to review

If you wish to submit a suggestion of an article or claim to review, please use this online form. Please note that we focus on reviewing claims and articles that are scientifically verifiable and that reach large audiences.

The editor in charge of overseeing the review process searches the scientific literature for relevant evidence to assess the claims’ credibility and invites relevant experts to contribute to the review. Editors evaluate claims based on trustworthy sources and provide citations for supporting evidence that they use in a review, including sufficient context and avoiding cherry-picking. Where appropriate and possible the editor reaches out to the person who made the claim. Notably when there is reasonable doubt over the interpretation of the claim.

• Read more about Science Feedback’s editorial standards.

Contributing scientists (reviewers) are invited to comment on the claims to assess their credibility and/or to indicate whether the facts underlying the reasoning are consistent with up-to-date scientific knowledge. Reviewers should represent the state of knowledge in the scientific literature, using strongly supported scientific theories and observations as references, and refrain from pointing to partial, isolated or weakly supported findings.

• Read more about Science Feedback’s community standards.

The primary objective of editors and reviewers should be to check the scientific accuracy of the facts and assumptions on which the claims are based. However, their contributions need not be limited to fact-checking and could extend to:

1. Adding Relevant Information: Signaling additional resources or information related to the text, providing context, or indicating a perspective based on a scientific publication or clinical experience.
2. Evaluating Scientific Reasoning: Misleading arguments are often based not on erroneous factual propositions, but on flawed/misleading reasoning or hidden and weakly supported assumptions.

Based on the evidence collected and feedback from scientists, the editor writes an article assessing the credibility of the claim or article. The review discusses relevant credibility criteria among the following:

1. Factual Accuracy. Does the article build on scientifically accurate information and solid evidence?
2. Scientific understanding. Beyond accurately describing basic observations, does the claim/article correctly interpret these observations and correctly state what’s currently known/unknown about the phenomenon discussed? Does it understand or misinterpret the significance of the observations?
3. Context and limitations. If the claim/article is discussing new research, is it adequately discussing pre-existing relevant studies, and are the new results not overly simplified, overstated or over-hyped?
4. Logic/Reasoning. Do the conclusions follow from the evidence? Are there gaps or leaps in the reasoning?
5. Precision/Clarity. Does the article precisely refer to concepts as they are used by the scientific community? Or is there some confusion which might mislead the reader?
6. Sources Quality. Does the claim or article rely on adequate and credible sources (use of relevant/independent experts, studies) to back up important claims? Or is it relying on experts with a conflict of interest?
7. Fairness/Objectivity. Does the article depict a complete or imbalanced view of the relevant science? Does it give precedence to certain sources without an appropriate reason for doing so?

Prior to publication, all of our articles are reviewed by at least one other editor who challenges the statements made and pinpoints where additional clarity and references are needed.

To help readers quickly understand the conclusion reached, we start claim reviews with a “key takeaways” section, which lays out what one needs to know about the topic and article reviews with a summary of the most salient points brought by the scientists.

We also issue a credibility rating assessing the overall credibility of the claim or article analyzed. Scroll down to read more about our claim review verdicts and our article review ratings.

The review is promoted on our website, across our social platforms, and shared with media and scientific partners. When possible, the Science Feedback editor provides feedback to the journalists and/or editors of the outlet of the original article reviewed.

Corrections

We aim for our reviews to be as accurate and up-to-date as possible. If we discover a mistake has been made, we will correct it as soon as possible and a note will be added on the original item. If you think we’ve made an error or missed some relevant information, contact us.

• Read our full corrections policy

When assessing articles, we invite scientists to provide an assessment of its overall scientific credibility in the form of a numerical rating and a set of credibility keywords. The following criteria are used to help guide the evaluation:

Suggested guidelines for the overall scientific credibility rating

Remember that we do not evaluate the opinion of the author, but instead the scientific accuracy of facts contained within the text, and the scientific quality of reasoning used.

• +2 = Very High: Little to no inaccuracies, fairly represents the state of scientific knowledge, contains appropriate references or links. The article provides insights to the reader about relevant science, mechanisms and implications, as well as limitations and important unknowns surrounding the evidence.
• +1 = High: The article does not contain major scientific inaccuracies and its conclusion follows from the evidence provided. While more detail would have been useful, readers are still accurately informed of the science.
• 0 = Neutral: No significant errors, but not enough insight either to inform the reader. (Ex: Article does not misstate findings from observational study but does not point out experimental research is needed to confirm findings; article doesn’t point out that unpublished research findings aren’t peer-reviewed…)
• -1 = Low: The article contains significant scientific inaccuracies or misleading statements.
• -2 = Very Low: The article contains major scientific inaccuracies for key facts supporting argumentation, and/or omits important information, and/or presents logical flaws in using information to reach conclusions.
• n/a = Not Applicable: The article does not build on scientifically verifiable information (e.g. it is mostly about policy, politics or opinions).

The final Science Feedback rating is the average of all the reviewers’ ratings; in cases when reviewers largely disagree no rating is issued and the review is left as ‘debated’.

Credibility criteria, definition of the keywords

• Accurate: free from factual errors, describes reality in a way that is consistent with available data/observations.
• Inaccurate: contains statement of fact in direct contradiction with available observations/data.
• Insightful: offers a deep understanding of the issue based on accurate information and proper context that clarifies the implications of observations.
• Misleading: offers an incorrect impression on some aspect(s) of the science, leaves the reader with false understanding of how things work, for instance by omitting necessary background context.
• Biased: holds some ideas (persons) as true (right) without proper justification, lack of objectivity, ideological.
• Unbiased: not biased, impartial, weighs evidence for/against ideas
• Sound reasoning: conclusion follows from the argument presented.
• Flawed reasoning: conclusion does not follow from the argument presented.
• Cherry-picking: highlights only a subset of all the available relevant evidence that seem to confirm a particular conclusion, ignoring a significant portion of evidence that would contradict it.
• Exaggerating: overstates / exaggerates the significance of some findings. (e.g. claims that a new scientific study overturns previous knowledge while it is an incremental update)
• Inappropriate sources: relies on low credibility sources, provides no or insufficient evidence in support of claims made.
• Imprecise/Unclear: uses ill-defined terms or lacks specifics so that one cannot unambiguously know what is meant without making additional unstated assumptions.
• Clickbait headline: article does not appropriately support its title.
• Conflates facts and opinions: presents opinion as fact or fact as opinion.
• Lack of context: lack of observations or explanations that would change the reader’s takeaway.
• Undisclosed conflict of interest: article fails to disclose a conflict of interest with a strong likelihood of influencing a source’s conclusions.
• Misrepresentation of sources (strawman): Substitutes a misrepresentation of a source’s conclusion for its actual conclusion, often in order to make it easier to discredit the idea of an “opponent.”
• Overstates scientific confidence: Presents a conclusion as conclusive while the hypothesis is still being investigated and there remains genuine scientific uncertainty about it.

Claim review verdicts

Background

Based on Climate Feedback and Health Feedback article reviews, the dimensions of information credibility scientists have identified in scientific reporting cluster around the following categories:

• whether facts are accurate,
• whether explanations convey correct interpretation and understanding of the facts with sufficient context, and
• whether the conclusions reached follow sound logic.

Informative reporting also relies on:

• the use of clear and precise language, and not ambiguous or vague claims,
• the use of adequate sources to back up important claims (“experts”, references…), and
• following an unbiased approach to guarantee objectivity.

see Science Feedback’s framework to evaluate the scientific credibility of information at the article level.

Verdicts

We rate the scientific credibility of claims following our scale of scientific credibility. In our climate and health claim reviews, the background color of the tag indicates the scientific credibility level according to this color code.

Our “verdict tags” also feature a word summarizing the reason that the claim earned its credibility rating. Here is how we rate claims:

• Cases where a claim is of “Very High” credibility:

If the claim is a statement of fact, it is tagged as “Accurate” when it describes an observation in a way that is consistent with available data and does not leave out any relevant element of context. If the claim is an explanation of the causes of an observation (aka a “theory” or an “hypothesis” in science), it is deemed “Correct” when it has been well tested in scientific studies and generates expected observations that are confirmed by actual observations.

Tags used in this category:

Note that in science a theory cannot be “proved” to be correct, in the mathematical sense of the term. The words “Correct” and “Accurate” are often both used to qualify an explanation, but for the sake of avoiding ambiguity we reserve the use of the words ”(in)accurate” for statements of facts and ”(in)correct” for explanations, hypotheses, and theories.

• Cases where a claim is of “High” credibility:

A statement of fact is deemed “Mostly Accurate” if it needs some clarification or additional information to be fully accurate. An explanation is deemed ‘Mostly Correct’ if it presents a theory that is well tested in scientific studies, but its formulation in the claim might overstate the confidence scientists actually have in the theory or slightly distort what can be predicted based on the theory. Tags used in this category:

• Cases where a claim is of “Neutral” credibility:

A claim is characterized as “Neutral” if it leaves out important information or is made out of context (“Lacks Context”). For instance, a claim would be tagged “Partially Correct” if it significantly overstates scientific confidence in a theory, or “Imprecise” if it uses ill-defined terms or lacks specifics so that one cannot unambiguously know what is meant without making additional unstated assumptions. Tags used in this category:

• Cases where a claim is of “Low” credibility:

A claim is deemed of “Low” credibility when it is made without backing from an adequate reference or if the available evidence does not support the statement (tagged as “Unsupported”). If a claim contains an element of truth but leaves the reader with a false understanding of reality, for instance by omitting critical background context, it would be tagged as “Misleading”. Tags used in this category:

• Cases where a claim is of “Very Low” credibility:

A claim is deemed of “Very Low” credibility when it is clearly wrong—for instance, if it makes a statement of fact in direct contradiction with available data (tagged as “Inaccurate”), or if it provides an explanation or a theory whose predictions have been invalidated (tagged as “Incorrect”). Tags used in this category:

Process for deciding on a verdict

The final ruling regarding the verdict attributed to the claim is made by a Science Feedback editor based on suggestions by the scientists contributing to the review.

Details

The “verdict details” section provides an explanation of the issue(s) raised by scientists—notably when a statement is implicitly or explicitly composed of several sub-claims. Below is a list of issues that have been raised in reviews along with a brief definitions.

Issues related to factual accuracy

• Factually inaccurate: A statement of fact in direct contradiction with available observations/data.
• Conflates factual statement and opinion: Presents opinion as fact or fact as opinion.

Issues related to explanations

• Correct: The theory/hypothesis is consistent with available data and has not been disproven.
• Misleading: Leaves the reader with a false or poor understanding of how things work.
• Misrepresents a complex reality: Fails to recognize that an observation can be influenced by more than one factor.
• Fails to grasp significance of observation: Uses an observation in support of a conclusion that it does not support.

  Issues related to logic

• Flawed reasoning: Reasoning is flawed if conclusions do not follow from the premises.

  Issues related to context

• Lack of context: The claim lacks elements of context (observations or explanations) that would change the reader’s takeaway.
• Cherry-picking: The claim depends on highlighting only a subset of all the available relevant evidence.

  Issues related to representation of the scientific process

• Overstates scientific confidence: Presents a conclusion as conclusive while the hypothesis is still being investigated and there remains genuine scientific uncertainty about it.
• Overstates the scientific impact of a finding: For instance, claims that a new scientific study overturns all previous knowledge when, in reality, it is just an incremental update.

  Issues related to sources

• Inadequate support: Reference used to support the claim is non-existent, of low scientific credibility or insufficient (according to the principle that extraordinary claims require extraordinary evidence).
• Misrepresents source (Strawman): Substitutes a misrepresentation of a source’s conclusion for its actual conclusion, often in order to make it easier to discredit the idea of an “opponent”.

  Issues related to precision or clarity of language

• Lacks specifics: The claim is too broad or vague, or lacking supporting details, to be clearly verifiable.
• Imprecise: Uses a scientific term in a way that does not refer to the concept in the same way it is used in science.

---

Note: This typology is based on concrete examples that have been observed in Climate Feedback and Health Feedback reviews so far. It can be updated as we encounter new examples.