WEIRD Audits? Research Trends, Linguistic and Geographical Disparities in the Algorithm Audits of Online Platforms -- A Systematic Literature Review

TL;DR

WEIRD Audits? systematically analyzes 176 public-facing algorithm audits to reveal strong Western, English-language, and WEIRD biases in platforms studied and attributes examined. Using a PRISMA-based review and an expanded coding scheme, the study shows distortion and discrimination as dominant issues, with search and Google/YouTube receiving outsized attention, while non-Western platforms and languages remain underexplored. The work highlights gaps in geographic, linguistic, and attribute diversity and offers concrete recommendations for broader global collaboration, methodological inclusivity, and policy relevance. Overall, the paper underscores that generalizing audit findings across contexts requires more diverse datasets, platforms, and analytical lenses to ensure fair and globally applicable insights.

Abstract

The increasing reliance on complex algorithmic systems by online platforms has sparked a growing need for algorithm auditing, a methodology evaluating these systems' functionality and impact. In this paper, we systematically review 176 peer-reviewed online platform-focused algorithm auditing studies and identify trends in their methodological approaches, the geographic distribution of authors, and the selection of platforms, languages, geographies, and group-based attributes in the focus of the reviewed research. We find a significant skew of research focus towards few online platforms, Western contexts, particularly the US, and English language data. Additionally, our analysis indicates a tendency to focus on a narrow set of group-based attributes, often operationalized in simplified ways, which might obscure more nuanced aspects of algorithmic bias and discrimination. We provide a clearer understanding of the current state of the online platform-focused algorithm auditing and identify gaps to be addressed for a more inclusive and representative research landscape.

Figures (6)

• Figure 1: Number of audits examining each problem type by year
• Figure 2: Number of audits examining each specific problem type by year
• Figure 3: Number of audits examining each domain by year
• Figure 4: Number of audits examining each of the top 10 most frequently audited platforms by year
• Figure 5: Countries color-coded by the N of auditing studies that included data from a given country context.