Algorithmic Behaviors Across Regions: A Geolocation Audit of YouTube Search for COVID-19 Misinformation Between the United States and South Africa

TL;DR

This study conducts the first large-scale geolocation-based audit of YouTube search for COVID-19 misinformation across the US and SA, using sock-puppet bots across three geolocations per country and 48 queries over 10 days to collect 915K results. It develops a ground-truth labelled set of 3,075 videos and trains multiple classifiers, finding DeBERTa-v3-large performs best for English content with an accuracy of about 0.85; non-English videos are handled separately. The results reveal that YouTube search behaves differently by geolocation, with SA users encountering more misinformative top-tier results, and several topics (e.g., 5G, Bill Gates claims, and vaccine content) showing regional disparities, while some topics are mitigated by YouTube’s moderation in the US. The authors argue for regionally consistent algorithmic regulation and highlight the practical health risks of misinformative SERPs, especially in the Global South, while also discussing methodological challenges in Global South audits and suggesting directions for future work. Overall, the paper contributes a rigorous cross-region methodological framework, a rich labeled dataset, and empirical evidence of geographic inequities in misinformation exposure on YouTube, with implications for platform governance and public health policy.

Abstract

Despite being an integral tool for finding health-related information online, YouTube has faced criticism for disseminating COVID-19 misinformation globally to its users. Yet, prior audit studies have predominantly investigated YouTube within the Global North contexts, often overlooking the Global South. To address this gap, we conducted a comprehensive 10-day geolocation-based audit on YouTube to compare the prevalence of COVID-19 misinformation in search results between the United States (US) and South Africa (SA), the countries heavily affected by the pandemic in the Global North and the Global South, respectively. For each country, we selected 3 geolocations and placed sock-puppets, or bots emulating "real" users, that collected search results for 48 search queries sorted by 4 search filters for 10 days, yielding a dataset of 915K results. We found that 31.55% of the top-10 search results contained COVID-19 misinformation. Among the top-10 search results, bots in SA faced significantly more misinformative search results than their US counterparts. Overall, our study highlights the contrasting algorithmic behaviors of YouTube search between two countries, underscoring the need for the platform to regulate algorithmic behavior consistently across different regions of the Globe.

Figures (13)

• Figure 1: Pipeline Overview. Sock-puppet bots emulating real-world users utilized the curated search queries to gather YouTube search engine result pages (SERPs) from geolocations in the United States (US) and South Africa (SA). After training and employing a classifier to scale the video labeling process, we compared the prevalence of COVID-19 misinformation in SERPs between the two countries.
• Figure 2: Distribution of the mean misinformation bias scores for the top-10 to top-50 videos in SERPs across the US and SA geolocations. These scores were computed considering the top number of videos (N) in the SERPs. Scores greater than 0 indicate that the SERPs lean toward supporting misinformation, while scores below 0 suggest SERPs lean toward opposing misinformation. Higher scores reflect a greater prevalence of misinformative videos. To compare scores between the US and SA at each level, we performed Mann-Whitney U Tests with test statistics in Appendix Table \ref{['tab:top10-50-mann-whitney']}. Note that: *p$<$0.05; **p$<$0.01; ***p$<$0.001.
• Figure 3: For each topic, we indicate the average misinformation bias scores of the top-10 search results between the US and SA (heatmap) and conduct a Mann-Whitney U Test to compare these bias scores between the two countries. We denote the p-value (p), Mann-Whitney effect size (r), U-value, and the mean rank difference. For example, for the "5G Claims" topic, SA $>$ US in the "Mean Rank Diff." column indicates that bots in SA received more misinformative videos in the top-10 search results than bots in the US. Note that: *p$<$0.05; **p$<$0.01; ***p$<$0.001.
• Figure 4: Mean misinformation bias scores for the top-10 search results across all 8 topics and 4 search filters between the US and SA. Note that "Relevance" is YouTube's default sorting filter for search results.
• Figure 5: For each search filter, we indicate the average misinformation bias scores of the top-10 search results between the US and SA (heatmap) and conduct a Mann-Whitney U Test to compare these bias scores between the two countries. *p$<$0.05; **p$<$0.01; ***p$<$0.001.