Despite Community Notes, most content reviewed by fact-checkers in Europe goes unaddressed on X/Twitter

Nov 21 2024 update: an earlier version of this article mistakenly stated that 67.2% of tweets displayed no visible moderation action, whereas the correct figure is 68.8%.

Executive summary

Looking at pan-European data from the European Fact-Checking Standards Network (EFCSN)’s database of content reviewed around the 2024 European Parliament elections, we find that most of the content on X (formerly Twitter) that fact-checkers found to be false or misleading had no visible sign of moderation action.

Specifically, out of the 894 tweets that European fact-checkers from 22 countries identified as containing misinformation and uploaded to the EFCSN database:

• 19.5% have been deleted from the platform,
• 11.7% had a Community Note and/or another “out of context” warning label,
• 68.8% had received no visible action.

While X is presenting the crowd-sourced Community Notes system as a substitute for professional fact-checking, our results show that in a European context, Community Notes (and more broadly, X’s content moderation systems) fail to identify the posts flagged by professional fact-checkers, leaving misleading and oftentimes highly-viral content unaddressed. 

Looking at Community Notes more broadly, further doubts are raised regarding their overall adequacy in labeling misleading or false content at the necessary scale.

First, when looking at EU countries that account for 60 million platform users, only 51,640 Community Notes (roughly one in 500,000 tweets) were displayed in the first 10 months of 2024 (a period which includes the EP elections). This volume of Community Notes is likely to be orders of magnitude less than the prevalence of misleading information on the platform.

Second, we see extreme disparity in the density of Community Notes density across EU Member States. The highest-density country (Denmark, at 2.44 notes per 1,000 users over a 10-month period) has 32 times more Community-Notes-per-active-user than the lowest-density country (Hungary, at 0.08). This imbalance suggests that, at least in some Member States, Community Notes offer inadequate coverage.

---

Are Community Notes a suitable substitute for professional fact-checking ?

Twitter’s BirdWatch, which was rebranded into X’s Community Notes, has been presented by platform owner Elon Musk as a “game changer for combating wrong information” and has been contrasted with the traditional fact-checking process, in which specialized organizations assess the factual accuracy of a claim.

The Elections 24 check database

In the run-up to and the aftermath of the 2024 European Parliament election, the European Fact-Checking Standards Network (EFCSN) created a common database for its 40+ members to contribute data about the claims they reviewed and the surfaces on which those claims had appeared. The database contains over 3,000 verification articles spanning 35 European countries.

From this database, we extracted all tweets that had been rated by fact-checkers as containing misinformation, yielding 894 unique pieces of content reviewed by fact-checkers from 22 countries. 

Note that this figure does not mean that there were only 894 pieces of misinformation content on X/Twitter, nor that fact-checkers did not identify more, but it only reflects the number of tweet URLs uploaded by EU fact-checkers in this specific database. A more thorough analysis could look at all the occurrences and variations on the platform of the misleading claims identified by European fact-checkers.

Reviewing X’s moderation actions

On Nov. 7 2024, we visited each of the 894 tweet URLs and coded whether:

• The tweet had been deleted,
• The tweet had a Community Note attached to it
• The tweet had any other credibility-related warning label (such as a warning that the content was AI-generated).

Results

Status	Number of tweets	Total tweet views	Average tweet views
Deleted/Inaccessible	174 (19.5%)	n/a	n/a
Community Note only	81 (9.1%)	49,622,938	612,628
“Out of context” label only	19 (2.1%)	4,120,779	216,883
Community Note + “Out of context” label	5 (0.6%)	13,173,000	2,634,600
No visible moderation action	615 (68.8%)	565,698,447	919,835

These results suggest that, for all their possible merits, neither the Community Notes program nor X’s other content moderation tools are adequately identifying content shown to be misinformation by fact-checkers.

We see two non-mutually exclusive explanations for this discrepancy:

• Community Notes users do not request reviews on the same content that fact-checkers deem checkworthy,
• The Community Notes algorithm choosing whether to display a Note on a tweet requires cross-partisan agreement on the usefulness of the Note, contrary to fact-checkers whose ratings are based on the underlying content’s factual accuracy. Consequently, if a Community Note cannot bridge the partisan divide, it will never be shown, even if the content contains misinformation.

Do Community Notes offer sufficient coverage across EU Member States ?

Assuming a comparable prevalence of mis- and disinformation across EU countries on Twitter/X, a well-functioning Community Notes system should offer a comparable density of Community Notes per X/Twitter user across EU Member States.

To check this hypothesis, we downloaded from the official repository (on Nov 7 2024) all Community Notes published since January 1 2024, then assigned a language to each Note (using the lingua Python library). For each language, we were therefore able to obtain a number of notes published since the beginning of the year.

We then collected the number of official Twitter/X users in each EU country from the platform’s official DSA transparency report. 

After filtering out the EU official languages for which the majority of speakers are not in the EU (English, French, Portuguese, Spanish), we paired each language to the EU Member State country where it is the primary language (ignoring the issue of minority languages for simplicity). We then computed a number of Community Notes per platform user metric for each country.

Language	Number of Community Notes published (Jan 1 – Nov 7 2024)	Number of Twitter/X users	Notes per 1,000 users	Notes per 1,000 users (percentage of max)
Danish	2,867	1,173,182	2.44	100.0%
Slovak	662	533,508	1.24	50.8%
German	21,544	18,363,782*	1.17	48.0%
Polish	7,571	9,113,116	0.83	34.0%
Czech	1,686	2,058,333	0.82	33.5%
Italy	6,451	8,198,871	0.79	32.2%
Dutch	6,302	8,371,941	0.75	30.8%
Finland	1,397	2,317,605	0.60	24.7%
Slovenian	253	446,017	0.57	23.2%
Estonian	162	294,843	0.55	22.5%
Swedish	1,405	2,660,113	0.53	21.6%
Croatia	235	768,561	0.31	12.5%
Lithuanian	162	556,826	0.29	11.9%
Romanian	474	1,905,881	0.25	10.2%
Latvian	67	439,888	0.15	6.2%
Greek	248	1,901,470	0.13	5.3%
Bulgarian	89	705,107	0.13	5.2%
Hungarian	95	1,239,000	0.08	3.1%
Total	51,670	61,048,044	0.84	34.6%

First, we notice that the absolute number of Community Notes in these Member States appears particularly low: a back of the envelope calculation suggests that, in those EU countries, the prevalence of Community Notes is about 1 in 500,000 tweets (61 million users in our sample of EU countries represent roughly 16.5% of the 370 million Twitter users worldwide, hence 16.5% of of the estimated 500 million worldwide daily tweets so approximately 25 billion tweets over the period Jan 1 – Nov 7 2024, covered by 51,670 Community Notes).

To the best of our knowledge, no reliable statistic on the prevalence of misleading content on the platform exists. We therefore do not have a robust baseline against which to check the coverage of Community Notes.

However, proxy indicators such as the University of Michigan’s Iffy Quotient or the EU Code of Practice of Disinformation’s Structural Indicator on Discoverability suggest a prevalence of disinformation orders of magnitude greater than 1 in 500,000 tweets.

A further point of comparison is Facebook reporting over 23.3 million pieces of content labeled as misinformation in these same countries in the first half of 2024 (X/Twitter’s 51,670 Community Notes cover the period January 1 – Nov. 7).

Second, unless there was a strong difference in the prevalence of misinformation on X/Twitter in different Member States (which the Structural Indicator on Discoverability does not suggest), we do not see any evident exogenous explanation for the extreme discrepancies between per-user Community Notes coverage across Member States. This casts further doubt as to whether Community Notes is adequately tailored to meet the challenge of misinformation on the platform.

Discussion and limitations

This study is not an exhaustive evaluation of all aspects of Twitter/X’s misinformation content moderation system, and particularly of its flagship Community Notes program. For instance, we do not look at Community Notes’ time-to-publication, at their impact on post influence (1, 2) or acceptability by users, their susceptibility to manipulation, or at their factual accuracy. All of these factors are theoretically important in evaluating whether the system meets its intended objectives.

However, absent sufficient coverage of misinformation posts by those Community Notes, any debate about their effectiveness is moot. Our results quite simply suggest that, in the EU, the vast majority of tweets containing misinformation do not get addressed.