Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs

TL;DR

This work tackles the problem of auditing fairness in black-box LLMs under strict query budgets. It introduces BAFA, a bounded active fairness auditing framework that treats auditing as uncertainty estimation over a target metric (such as the difference in ROC AUC, ΔAUC) and uses constrained empirical risk minimisation over a version space of surrogates to produce tight uncertainty intervals. BAFA actively selects queries to shrink the interval width, yielding substantial query-efficiency gains demonstrated in two real-world case studies (CivilComments and Bias-in-Bios) with up to ∼40× fewer queries and improved stability over time. While BAFA offers a practical, independent auditing mechanism for continuous evaluation, it also discusses limitations (surrogate choice, computational intensity) and emphasizes cautious interpretation of uncertainty bounds as operational certificates rather than definitive guarantees of overall safety or fairness.

Abstract

Large Language Models (LLMs) exhibit systematic biases across demographic groups. Auditing is proposed as an accountability tool for black-box LLM applications, but suffers from resource-intensive query access. We conceptualise auditing as uncertainty estimation over a target fairness metric and introduce BAFA, the Bounded Active Fairness Auditor for query-efficient auditing of black-box LLMs. BAFA maintains a version space of surrogate models consistent with queried scores and computes uncertainty intervals for fairness metrics (e.g., $Δ$ AUC) via constrained empirical risk minimisation. Active query selection narrows these intervals to reduce estimation error. We evaluate BAFA on two standard fairness dataset case studies: \textsc{CivilComments} and \textsc{Bias-in-Bios}, comparing against stratified sampling, power sampling, and ablations. BAFA achieves target error thresholds with up to 40$\times$ fewer queries than stratified sampling (e.g., 144 vs 5,956 queries at $\varepsilon=0.02$ for \textsc{CivilComments}) for tight thresholds, demonstrates substantially better performance over time, and shows lower variance across runs. These results suggest that active sampling can reduce resources needed for independent fairness auditing with LLMs, supporting continuous model evaluations.

Figures (8)

• Figure 1: Bounded Active Fairness Auditing (BAFA). Upper and lower bounds on the fairness metric converge as queries accumulate. BAFA to maximally shrink the uncertainty interval between bounds.
• Figure 2: BAFA Pipeline in more detail. In every turn we sample $k$ samples. First, we query the black-box LLM with a stratified seed set from our dataset (1). Then, we calculate the estimated fairness measure (2) and do constraint optimisation with BERT surrogates (3) to get lower and upper fairness bounds. Based on the calculated scores for each $x\in D$ from the upper and lower models (4), BAFA selects queries (5) which shrink the distance between the lower and upper in high-disagreement regions, leading to faster and more stable convergence.
• Figure 3: Active auditing methods perform more query-efficient ant stable over 20 Civil Comments seeds. BAFA methods (solid) converge significantly faster than baseline sampling strategies (dotted). Shaded areas indicate 95% confidence intervals across seeds and demonstrate that BAFA methods show substantially reduced variance compared to baseline methods.
• Figure 4: Active auditing methods perform even with large parameter spaces with GPT-4.1-mini as black-box. Similarly, to Fig. \ref{['fig:plot_time_1']}, BAFA methods converge significantly faster than baseline sampling strategies and show substantially reduced variance compared to baseline methods. However, a much bigger variance and worse performance are visible for the first 100-120 queries, probably related to the model mismatch.
• Figure 5: Relationship between BAFA’s uncertainty bound width and the true absolute estimation error of $\Delta$AUC in first 1k queries for Civil Comments. Each point corresponds to an audit during active querying. Bound width strongly correlates with actual error for both BO and disagreement-based selection.