AI auditing: The Broken Bus on the Road to AI Accountability

TL;DR

This paper interrogates the AI auditing landscape, showing that while audits are proliferating across academia, law, journalism, civil society, and government, few translate into concrete accountability outcomes. It presents a taxonomy of audit types—product/model/algorithm, data, ecosystem, and meta-commentary—and analyzes how design, context, and stakeholder dynamics shape effectiveness. The authors synthesize 341 academic studies and a broad set of non-academic audits to identify practices associated with greater accountability, such as ecosystem-wide scope, explicit harms discovery, stakeholder engagement, and transparent communication. Key findings reveal power asymmetries, limited cross-domain impact for many product/data audits, and stronger policy or legal changes driven by journalism and government audits. The work provides practical guidance for researchers and policymakers to augment audit impact, advocate for broader governance, and acknowledge the limits of audits within a holistic AI accountability framework.

Abstract

One of the most concrete measures to take towards meaningful AI accountability is to consequentially assess and report the systems' performance and impact. However, the practical nature of the "AI audit" ecosystem is muddled and imprecise, making it difficult to work through various concepts and map out the stakeholders involved in the practice. First, we taxonomize current AI audit practices as completed by regulators, law firms, civil society, journalism, academia, consulting agencies. Next, we assess the impact of audits done by stakeholders within each domain. We find that only a subset of AI audit studies translate to desired accountability outcomes. We thus assess and isolate practices necessary for effective AI audit results, articulating the observed connections between AI audit design, methodology and institutional context on its effectiveness as a meaningful mechanism for accountability.

Figures (3)

• Figure 1: Audit studies considered in our survey, classified by their scope.
• Figure 2: Number of collected academic audit studies published in each year, grouped by manually labelled audit type. Studies that involve multiple audit types are counted only under the best-fitting category.
• Figure C-I: Number of collected academic audit studies published in each year, grouped by publication source.

Theorems & Definitions (3)

• Definition 1
• Definition 2
• Definition 3