Consensus Credibility Scores: technical appendix

See main project description here

1- 2023 Academic Ranking of World Universities

Description: the list offers an annual ranking of the world’s leading 1,000 universities, computed on the basis of 6 criteria of academic excellence. Accessible here.

Coverage: 892 domains (some universities’ domains are not published by ARWU) 

Mapping onto the common zero-to-one scale: all entries received the maximal score of +1.

2- Ad Fontes Media

Description: AdFontes Media publishes assessments of news sources’ reliability and political bias, on the basis of an analysis of a sample of content produced by the news source. Its coverage focuses primarily on the United States. AdFontes Media’s ratings are available here.

As the full list of ratings is not publicly available, the public subset curated by Lin et al (2023) was used.

Coverage: 284 domains

Mapping onto the common zero-to-one scale: quantile transformation of the ‘afm_rely’ score column in Lin et al (2023). (lowest-ranked received zero, highest-ranked received 1)

3- Bufale.net

Description: Bufale.net is an Italian website dedicated to debunking misinformation and verifying online news stories. Their ‘blacklist’ can be accessed here.

Coverage: 80 domains

Mapping onto the common zero-to-one scale: all domains on the list received a score of 0.

4- Bufalopedia

Description: Bufalopedia is an Italian-language initiative aimed at debunking falsehoods spreading online. Their list of ‘hoax-generating sites’ can be accessed here.

Coverage: 53 domains

Mapping onto the common zero-to-one scale: all domains on the list received a score of 0.

5- Butac.it

Description: BUTAC (“Bufale un tanto al chilo”) is an Italian fact-checking initiative operating a website and various social media channels. Their ‘blacklist of the Italian web’ can be accessed here.

Coverage: 233 domains

Mapping onto the common zero-to-one scale: all domains on the list received a score of 0.

6- Center for Media, Data and Society (CMDS)

Description: The Central European University-affiliated CMDS (now closed, largely replaced by the Media and Journalism Research Center) produced a series of reports on misinformation, some of which included lists of low-credibility domains. The lists published in the reports for Serbia, Hungary and Slovakia were used. 

Coverage: 167 domains

Mapping onto the common zero-to-one scale: all domains on the lists received a score of 0.

7- Conspiracy Watch

Description: Conspiracy Watch started as an initiative tackling antisemitism and Holocaust denial, and gradually broadened its scope to include all topics of disinformation. It covers mainly French-speaking sources. The sources of disinformation identified by Conspiracy Watch are published here.

Coverage: 158 domains

Mapping onto the common zero-to-one scale: all domains on the list received a score of 0.

8- Courrier International

Description: Courrier International is a French weekly which collects and translates articles from other international newspapers, so as to offer a local point of view on events of international or regional significance. Courrier International tends to feature high-quality media. The sources used by Courrier International are listed here.

Coverage: 365 domains, corresponding to all sources for European countries listed in the Courrier International website, provided that they had been used by Courrier International at least once since 2015 and had an associated website.

Mapping onto the common zero-to-one scale: Courrier International does not offer a structured rating system but offers a natural-language description of the sources it uses. A human analyst read these notices and translated them onto the zero-to-one scale using the following rules:

• A baseline of 0.75: since Courrier International leans heavily towards using reliable media, a score of 0.75 was assigned to the source if the description did not include any other information relative to the source’s credibility.
• When the description used language suggesting outstanding reliability, (e.g. ‘newspaper of record’, ‘reputation for seriousness’…), a score of 1 was assigned.
• When the description mentioned a clear political alignment or bias from the source but did not question the source’s reliability, the rating was downgraded to 0.6.
• In cases when the description expressed some reservations about the quality of the source, a ‘penalty’ was added to the baseline, whose severity depended on the strength of the language used.

For instance, German tabloid Bild, whose description includes “[its] circulation is more impressive than its quality […]”, was assigned a score of 0.5. Gazeta Polska, described as showing “unwavering populism, extreme nationalism, and, regularly, anti-Semitic remarks” was assigned a score of 0.2.

9- European Media Systems Survey (EMSS)

Description: An academic project systemically assessing the media landscape in European countries, including local-expert surveys of media outlet’s credibility. The EMSS is available here.

Coverage: 162 domains

Mapping onto the common zero-to-one scale: linear mapping of the original 0-to-10 ‘accuracy’ column onto the zero-to-one scale.

10- Grinberg et al (2019)

Description: As part of an academic research project studying disinformation on Twitter in the context of the 2016 US elections, the authors established a list of ‘black sites’ propagating disinformation. The list is available here.

Coverage: 382 domains

Mapping onto the common zero-to-one scale: all domains on the list were assigned a score of 0.

11- International Fact-Checking Network (IFCN)

Description: A global forum for fact-checkers, hosted at the Poynter Institute. The IFCN publishes a stringent code of principles which is used as the global standard to certify fact-checkers’ non-partisanship, integrity and transparency. The list of verified signatories is available here.

Coverage: 165 domains

Mapping onto the common zero-to-one scale: all IFCN-certified fact-checkers verified (or whose certification is in the process of renewal) were assigned a score of +1.

12- Journalism Trust Initiative (JTI)

Description: A Reporters Without Borders initiative aimed at designing standards for transparency of media organization and professionalism of the editorial processes. The list of JTI-certified organizations is available here.

Coverage: 17 domains

Mapping onto the common zero-to-one scale: all JTI-certified media were assigned a score of +1 (self-declared JTI-compliant media that were not independently audited were not include)

13- Ketonen et al (2018)

Description: Preprint of an academic research paper, which aims to study the disinformation landscape on Facebook in Finland. As part of the research methodology, the authors identify reliable and unreliable sources of information to track. The preprint is available here.

Coverage: 26 domains

Mapping onto the common zero-to-one scale: all domains characterized as ‘real news websites’ were assigned a score of 0.85, while domains characterized as ‘disinformation websites’ were assigned a score of 0.15.

14- Konšpirátori.sk

Description: Konšpirátori.sk is a Slovak initiative that maintains a database of unreliable news sources in the Slovak and Czech online information spaces, with the primary purpose of allowing announcers to opt out of advertising on these websites. The project also maintains a gray list and a green list of websites, respectively covering borderline-unreliable and verified-information sources. Their lists are available here.

Coverage: 397 domains

Mapping onto the common zero-to-one scale: all domains on the main list were assigned a score of zero, all domains on the gray list were assigned a score of 0.3 and all domains on the green list were assigned a score of 0.8.

15- Le Décodex

Description: Fact-checking initiative from French newspaper Le Monde. The data is available here.

Coverage: 368 domains

Mapping onto the common zero-to-one scale: all domains in category ‘2’ were assigned a score of 0.

16- Lin et al (2023)

Description: Academic research article studying the correlation between different raters’ credibility scores for web domains. The article and associated datasets are available here.

Coverage: 11,519 domains

Mapping onto the common zero-to-one scale: quantile transformation of the ‘pc1’ column (lowest-ranked received zero, highest-ranked received 1)

17- Media Bias/Fact Check

Description: Media Bias/Fact Check is an online platform that evaluates the bias and factual accuracy of various media sources, drawing inter alia on observations of failed fact-checks. Although it covers primarily the US, other countries are also represented. Data is available here.

Coverage: 5,385 domains

Mapping onto the common zero-to-one scale: If the domain’s ‘Factual reporting’ category is ‘Very Low’, assign zero. If ‘Low’, assign 0.2. If ‘Mixed’, assign 0.4. If ‘Mostly Factual’, assign 0.6. If ‘High’, assign 0.8. If ‘Very High’, assign 1.

18- Nelež

Description: project launched by Czech media professionals to help demonetize sources of misinformation, drawing in part on data from Konšpirátori.sk and nfnz.cz. Data is available here.

Coverage: 21 domains

Mapping onto the common zero-to-one scale: score of 0 assigned to all domains on the list.

19- Nadační fond nezávislé žurnalistiky (NFNZ)

Description: project launched by Czech media professionals to raise the quality of the Czech-speaking information environment, notably by supporting high-quality independent journalism. Their Media Rating list is available here.

Coverage: 1,244 domains

Mapping onto the common zero-to-one scale: domains in the ‘junk’ or ‘conspiracy’ categories were assigned a score of zero, domains graded as ‘B-’ received a score of 0.25, domains graded as ‘B’ received a score of 0.4, domains graded as ‘B+’ received a score of 0.5, domains graded as ‘A-’ received a score of 0.75, domains graded as ‘A’ received a score of 1.

20- Ranking Web of World Research Centers

Description: ranking of academic research centers compiled by the Cybermetrics Lab, a research group falling under the umbrella of the CSIC, Spain’s public research network. Their ranking is available here.

Coverage: 6,246 domains

Mapping onto the common zero-to-one scale: all domains on the list assigned a score of 1.

21- Raskrinkavanje

Description: Raskrinkavanje is a fact-checking initiative covering multiple countries in the Western Balkans. Their ‘red flag’ list is available here and ‘high risk’ list is available here.

Coverage: 36 domains

Mapping onto the common zero-to-one scale: all domains on the ‘red flag’ list were assigned a score of 0, all domains on the ‘high risk’ list were assigned a score of 0.25.

22- Reuters Institute’s 2023 Digital News Report

Description: The Reuters Institute at the University of Oxford publishes a yearly Digital News Report covering most EU countries, as well as some large media markets outside Europe. The reports include audience surveys of the trustworthiness of a number of media outlets. The reports are available here.

Coverage: 211 domains

Mapping onto the common zero-to-one scale: all media that was reported as trusted by at least 50% of respondents in the trust survey and reported as ‘not trusted’ by less than 25% of respondents were assigned a score of 1. 

23- Sanchez et al (2022)

Description: An academic research paper reporting on the training of a Swedish-language automated classifier to identify reliable and unreliable sources of information, which includes a curated list of reliable and unreliable news sources. The paper is available here.

Coverage: 27 domains

Mapping onto the common zero-to-one scale: all domains in the ‘reliable’ list were assigned a score of 0.85 and all domains on the ‘unreliable’ list were assigned a score of 0.15.

24- Vsquare

Description: Central and Eastern European investigative journalism group VSquare published a report unearthing a list of outlets spreading falsehoods about the war in Ukraine, drawing on a process of automated methods for candidate identification and human verification for validation. The report is available here.

Coverage: 217 domains

Mapping onto the common zero-to-one scale: all domains on the list were assigned a score of 0.

25- Wikipedia blocked domains – France

Description: French-language Wikipedia contributors maintain a list of domains that cannot be linked to from Wikipedia, because they promote disinformation or are associated with spam. The list is accessible here.

Coverage: 1,136 domains

Mapping onto the common zero-to-one scale: all domains on the list were assigned a score of 0.

26- Wikipedia list of fake news websites – Philippines

Description: Wikipedia contributors covering the Philippines maintain a list of ‘fake news websites’. The list is accessible here.

Coverage: 152 domains

Mapping onto the common zero-to-one scale: all domains on the list were assigned a score of 0.

27- Wikipedia list of fake news websites – English

Description: English-language Wikipedia contributors maintain a list of ‘fake news websites’. The list is accessible here.

Coverage: 54 domains (not all outlets listed have associated domains)

Mapping onto the common zero-to-one scale: all domains on the list were assigned a score of 0.

28- Wikipedia discussions of sources’ quality

Description: In addition to the lists detailed above, some Wikipedia language communities maintain lists of domains whose appropriateness for use as a source on the encyclopedia has been frequently discussed. These lists offer a natural language summary of the discussions that were held with regards to a specific source. In some cases, they also offer a reliability categorization of the source. 

The discussion on English-language Wikipedia is available here, that on French-language Wikipedia is available here, that on Swedish-language Wikipedia is available here, and a broader international one is available here.

Coverage: 685 domains for the English-language one, 183 domains for French, 13 domains for Swedish (only domains that were not covered in the English-language list), 1,403 domains for the international one.

Mapping onto the common zero-to-one scale: while the different lists do not share the same format, the approach used to map them was common:

• If the list offered a categorization (reliable/unreliable/no consensus), the baseline was assigned to 0.875 for ‘reliable’ domains, 0.5 for ‘no consensus’ domains and 0.125 for ‘unreliable domains’.

• From this baseline, adjustments were made on the basis of the natural-language description, with the magnitude of the adjustment dependent on the strength of the language used.

For instance, a domain categorized as no consensus but described as ‘previously considered a self-published fringe source’ was brought down from the baseline of 0.5 to 0.35. Conversely, a source categorized as reliable and described as a “top tier peer-reviewed journal” was brought up from 0.875 to 1.

• In the absence of any natural-language description beyond the categorization, the source was assigned the relevant baseline’s score.

A- Sources

1- EUvsDisinfo

Description: Database of Russian-linked disinformation maintained by the European External Action Service. The list is accessible here.

Coverage: 14,710 URLs

Weighing URLs: a weight of 1 was assigned to all cases, as the original data source does not differentiate between types and severity of disinformation.

2- Fake News in French Dataset – FNFDS

Description: Volunteer project aimed at documenting misinformation on the French-language Open Web. The list is accessible here.

Coverage: 623 URLs

Weighing URLs: a weight of 1 was assigned to all cases, as the original data source does not differentiate between types and severity of disinformation.

3- Fake News Corpus Spanish

Description: Spanish-language dataset curated by researchers as part of a hackathon aimed at training machine learning models to detect fake news. The dataset is accessible here.

Coverage: 1,543 URLs

Weighing URLs: a weight of 1 was assigned to all cases, as the original data source does not differentiate between types and severity of disinformation.

4- Décodex

Description: Fact-checking initiative from French newspaper Le Monde. The data is accessible here.

Coverage: 13,242 URLs

Weighing URLs: URLs marked as ‘faux’ or ‘infondé’ were assigned a weight of 2, URLs marked as ‘à nuancer’, ‘contestable’ or ‘c’est plus compliqué’ were assigned a weight of 0.2, URLs marked as ‘vrai’ were assigned a weight of -2.

5- Open Feedback

Description: Science Feedback-operated initiative to collect data from fact-checks, including information on the URLs where fact-checked content has been published. The portal is accessible here.

Coverage: 39,243 URLs

Weighing URLs: URLs marked as ‘false’, ‘unsupported’ or similar were assigned a weight of 2, URLs marked as ‘missing context’ or similar were assigned a weight of 0.2, URLs marked as ‘true’, ‘explanatory’ or similar were assigned a weight of -2.

6- VoxCheck Ukraine

Description: IFCN-certified fact-checking organization VoxCheck maintains a database of URLs found to contain Russian-aligned narratives linked to the war in Ukraine. Project information is accessible here while the data is accessible here.

Coverage: 13,131 URLs

Weighing URLs: a weight of 1 was assigned to all cases, as the original data source does not differentiate between types and severity of disinformation.

B- URL-level data processing

For each source of URL-level credibility assessment, URLs pointing to a social media platform were removed from the sample, since assigning a uniform credibility score to a website populated with a wide variety of user-generated content would be meaningless.

Depending on the source of data, different rating scales were used. For instance, data sourced from Open Feedback usually comes with a rating on a scale that includes ‘False’, ‘Partly False’, ‘Misleading’,’ Missing Context’, ‘True’ whereas most other sources offer a binary disinformation/not-disinformation label. To reflect these nuances, a source-specific mapping to assign a score to each URL was effected and is detailed for each source above.

On the basis of this mapping, each URL was assigned a corresponding ‘URL disinformation score’. URLs were then deduplicated (in this case, the URL disinformation score was computed as the average of the URL disinformation scores of the duplicates). The domain was extracted from each URL, and the sum of all URLs’ disinformation score was computed for each domain, resulting in a ‘domain disinformation score’.

Since popular domains tend to publish more content and be subject to more scrutiny than their less-popular counterparts, regardless of their editorial standards . For instance, the BBC, despite being widely recognized as a generally trustworthy source, shows up over 100 times in our database, both due to the sheer number of content it publishes and to the fact that it sometimes features guests who are not bound by its editorial rules.

To account for this audience-size bias, the domain disinformation score was normalized by the monthly traffic to the website, as recorded by Similarweb. To limit noise, websites that had less than five URLs were dropped from the sample, resulting in a score for 1,571 domains.

As with the other domain-level scores described in I-, a mapper was then applied to transform the ‘domain disinformation score per million visitors’ into a score onto the common zero-to-one scale.

The mapper used was the following:

• domain disinformation score per million visitors < 0.05 : assign 1
• 0.05 <= domain disinformation score per million visitors < 0.1 : assign 0.75
• 0.1 <= domain disinformation score per million visitors < 0.25 : assign 0.625
• 0.25 <= domain disinformation score per million visitors < 0.45 : assign 0.5
• 0.45 <= domain disinformation score per million visitors < 0.6 : assign 0.375
• 0.6 <= domain disinformation score per million visitors < 1.5 : assign 0.25
• domain disinformation score per million visitors >= 1.5 : assign 0

This mapping scale was devised using the following method:

• Two experienced analysts from Science Feedback independently looked at a subset of the domain disinformation scores and devised independent mapping scales.
• A new subset of domain disinformation scores was given to the analysts, to test their mapping scales on.
• The two analysts discussed their results and agreed on a final scale.

Finally, the resulting zero-to-one domain-level credibility score was used as an input (similar to the domain-level scores listed in I-) to compute the domain’s average credibility score.