Use of information panels on YouTube videos containing misinformation – a ’quick check’ study
Executive summary
Contextual information panels containing links to authoritative sources on topics prone to disinformation (such as COVID-19 and vaccines or climate change) are the only mechanism cited by YouTube when it comes to directing users to authoritative information under Measure 22.7 of the EU’s Code of Practice of Disinformation.
To our knowledge, the effectiveness of these information panels has not been thoroughly investigated to date.
A quick-check study using a sample of 940 videos containing misinformation according to professional fact-checkers finds that:
- only 350 videos (37% of videos, 15% of views) displayed an information panel,
- one quarter of the videos for which an appropriate information panel does exist do not display the panel, suggesting an imperfect matching engine between content and information panels,
- the same generic panels are applied to a very wide variety of specific misinformation claims and narratives, casting strong doubt as to whether they are sufficiently well-targeted to entice users to click them.
- these doubts are further compounded by YouTube displaying the same panels next to high-credibility content on the same topics, further increasing the likelihood that users do not engage with them.
YouTube does not report any effectiveness metric such as click-through rates for these panels, despite it being a clear ask in SLI 22.7.1. of the Code of Practice.
Introduction
As part of its commitments under the EU’s Code of Practice on Disinformation, YouTube has pledged to “design and apply products and features (e.g. information panels, banners, pop-ups, maps and prompts, trustworthiness indicators) that lead users to authoritative sources on topics of particular public and societal interest or in crisis situations” (Measure 22.7).
In its latest transparency report under the Code, YouTube mentions a single mechanism under Measure 22.7: “panels [which] provide additional context, with each designed to help users make their own decisions about the content they find. These information panels will show regardless of what opinions or perspectives are expressed in a video. If users want to learn more, most panels also link to the third-party partner’s website”.
Importantly, the context boxes are displayed equally on videos promoting misinformation or videos containing authoritative information. Information quality is not a factor in choosing whether the panel will be displayed.
YouTube makes large-scale use of these panels, with over 4 billion impressions in the EU for the first half of 2023. These panels look as such:
Youtube describes these topical context information panels as “show[ing] basic background info, sourced from independent, third-party partners, to give more context on a topic. If [the viewer] want[s] to learn more, the panels also link to the third-party partner’s website.”
Given that these panels are framed by YouTube as the cornerstone of its efforts to link users to authoritative content on critical topics, we would expect their efficacy to be the subject of scrutiny by academics and civil society. To date, this has not been the case: a Google Scholar search for “YouTube ‘information panels’ OR ‘context panels’” yields no article in which they are a primary focus.
Using data from Open Feedback, a database of fact-checked information circulating online, we therefore set out to study which systemic insights, if any, could be drawn regarding the placement of these information panels next to videos containing misinformation.
The dataset
Open Feedback contains 3,547 unique URLs pointing to YouTube videos (not Shorts) that have been assessed by professional fact-checkers and found to contain misinformation.
Out of these 3,547 videos, 2,607 (73%) were not available anymore at the time of data collection (October 2023). In many cases, YouTube offers a reason as to the video’s removal. These removals suggest that many of the videos were found by YouTube to run afoul of their community guidelines. For instance, out of the five most frequent reasons stated for a video’s inaccessibility, three were directly linked to YouTube content moderation efforts:
- “This video is no longer available because the YouTube account associated with this video has been terminated.” (544 videos)
- “This video has been removed for violating YouTube’s Community Guidelines” (302 videos)
- “This video has been removed for violating YouTube’s Terms of Service” (13 videos)
In addition, 246 videos are now private, which could be a response by the user to a warning from YouTube.
| Reason or subreason given by YouTube | number of videos |
|---|---|
| This video is not available in your country | 854 |
| Video unavailable/No subreason | 627 |
| This video is no longer available because the YouTube account associated with this video has been terminated. | 544 |
| This video has been removed for violating YouTube’s Community Guidelines | 302 |
| Private video | 246 |
| This video has been removed for violating YouTube’s Terms of Service | 13 |
| This video has been removed for violating YouTube’s policy on harassment and bullying | 4 |
| This video has been removed for violating YouTube’s policy on spam, deceptive practices, and scams | 4 |
| This video has been removed for violating YouTube’s policy on violent or graphic content | 2 |
| This video has been removed by the uploader | 2 |
| This video has been removed for violating YouTube’s policy on hate speech. Learn more about combating hate speech in your country. | 2 |
| This video contains content from Red Square Group, who has blocked it on copyright grounds | 1 |
| Video unavailable/No subreason | 1 |
| No reason/No subreason | 1 |
| This video has been removed for violating YouTube’s policy on nudity or sexual content | 1 |
| This video is no longer available due to a trademark claim by a third party | 1 |
| This video is no longer available because the uploader has closed their YouTube account. | 1 |
| This video is no longer available due to a copyright claim by Andrei Trukhonovets | 1 |
Since these 2,607 videos are not available, we were unable to gather more granular information as to what topic they covered, although it is likely that a large proportion was removed due to violations of YouTube’s COVID-19-related community guidelines (before being broadened to its current Medical Misinformation Policy, YouTube’s policies specifically banned a number of claims related to COVID-19 and vaccines).
Excluding the unavailable videos, the resulting sample contained 940 videos, which have garnered a total of 1.1 billion views, or 1.2 million views per video on average.
Results
Takeaway 1 – Very few information panels
Of the 940 videos online, all of which contain misinformation according to fact-checkers, only 350 (37% of videos, 15% of views) displayed an information panel.
Three types of panels, about COVID-19 and climate change, accounted for 96% of all panels encountered in our dataset. These three panels read as follows:
- “COVID-19 vaccine – Learn about vaccine progress from the WHO” linking to https://www.who.int/emergencies/diseases/novel-coronavirus-2019/covid-19-vaccines (45% of panels)
- “COVID-19 Get the latest information from the WHO on coronavirus” linking to https://www.who.int/emergencies/diseases/novel-coronavirus-2019 (40% of panels)
- “[Context] Climate change – United nations ‘Climate change refers to long-term shifts in temperatures and weather patterns; mainly caused by human human activities, especially the burning of fossil fuels’” linking to https://www.un.org/en/climatechange/what-is-climate-change (11% of panels)
Takeaway 2 – Relevant information panels exist for a quarter of unlabeled videos but are not displayed
Two main hypotheses could explain the low proportion of videos that display an information panel next to them:
- YouTube doesn’t have an information panel covering the specific topic covered in the video,
- YouTube has an information panel covering the specific topic covered in the video but its –likely-automated– matching system doesn’t detect that the video is about the related topic.
To test which hypothesis is more likely, we took a random sample of 100 videos which didn’t contain an information panel. The videos were in English or French and focused on a single topic (we did not include hour long videos discussing several subjects). These 100 videos were then reviewed by a human analyst to see whether any of the panels identified above would be a good match.
Out of those 100, 26 videos contained misinformation about a topic for which YouTube already had a panel but did not display it. See figure 5 for examples.
Figure 5 – Examples of videos containing misinformation about COVID-19 but no information panels.
Takeaway 3 – The same panels are applied to videos making very different claims, raising doubts as to their relevance and effectiveness
A vast majority (96%) of the information panels found in the dataset revolve around COVID-19, COVID-19 vaccines and climate change.
On each of these topics, the specific misinformation narratives and claims found in YouTube videos vary widely (see Figure 6, 7, 8 for examples). Yet, regardless of the specifics of the misinformation they promote, YouTube shows the same information panel.
The fact that individual videos touching on very different topics are all ‘addressed’ using the same, very generic, resource raises strong doubts as to their ability to properly inform users on the specific misleading claims they have been exposed to and entice them to visit the authoritative source linked.
In addition, YouTube’s policy of displaying the panels regardless of the credibility of the videos means that videos from authoritative sources also display the same panels (see e.g. here, here or here), further increasing the likelihood that users will not consider these panels as bringing useful context to the misinformation they are exposed to on YouTube.
Conclusion
This quick check study suggests that the information panels applied by YouTube are highly unlikely to have the desired effect of connecting users to authoritative information on topics prone to disinformation. We find that:
- Few topics are covered by the information panels, resulting in most disinformation videos in our sample not displaying any panel;
- The matching system is imperfect, resulting in about a quarter of videos about a topic that could be covered by existing information panels not having one;
- The panels are extremely generic and do not match the specificity of the disinformation claims or narratives made in the video. For instance, it is doubtful that a user viewing a video about the Great Reset conspiracy will click on a panel linking to the WHO’s generic COVID-19 page.
- The panels are also applied next to authoritative content on these issues; so they offer no signal as to the content’s credibility to users. The most likely outcome in our view is that these panels end up being considered background noise by users.
Some of these doubts could be dispelled by YouTube providing data on the click-through rate for these information panels, as requested under Code of Practice on Disinformation Service Level Indicator 22.7.1. Unfortunately, YouTube has not offered this metric as of today.
