Pairwise Comparison for Bias Identification and Quantification
Fabian Haak, Philipp Schaer
TL;DR
This paper addresses the challenge of measuring linguistic bias with scarce gold labels by developing and evaluating cost-aware pairwise comparison methods. It advances theory (Elo and Bradley–Terry-based rating), optimization (streak/tail pruning and listwise ranking), and real-data application for bias detection and quantification, demonstrating robust, cost-efficient performance, particularly with listwise grouping and BT rating in LLM-driven setups. The work provides end-to-end evaluation, including simulations and benchmarks, and outlines an implementation blueprint for scalable bias annotation. Overall, it offers a principled, transparent framework for reproducible bias analysis in text. The practical impact lies in enabling efficient, auditable bias benchmarking and ongoing bias quantification across corpora and contexts.
Abstract
Linguistic bias in online news and social media is widespread but difficult to measure. Yet, its identification and quantification remain difficult due to subjectivity, context dependence, and the scarcity of high-quality gold-label datasets. We aim to reduce annotation effort by leveraging pairwise comparison for bias annotation. To overcome the costliness of the approach, we evaluate more efficient implementations of pairwise comparison-based rating. We achieve this by investigating the effects of various rating techniques and the parameters of three cost-aware alternatives in a simulation environment. Since the approach can in principle be applied to both human and large language model annotation, our work provides a basis for creating high-quality benchmark datasets and for quantifying biases and other subjective linguistic aspects. The controlled simulations include latent severity distributions, distance-calibrated noise, and synthetic annotator bias to probe robustness and cost-quality trade-offs. In applying the approach to human-labeled bias benchmark datasets, we then evaluate the most promising setups and compare them to direct assessment by large language models and unmodified pairwise comparison labels as baselines. Our findings support the use of pairwise comparison as a practical foundation for quantifying subjective linguistic aspects, enabling reproducible bias analysis. We contribute an optimization of comparison and matchmaking components, an end-to-end evaluation including simulation and real-data application, and an implementation blueprint for cost-aware large-scale annotation