Beyond Performance: Quantifying and Mitigating Label Bias in LLMs

TL;DR

The paper investigates label bias in large language models by quantifying predictions across 279 classification tasks and 10 LLMs, revealing substantial bias that persists despite scaling, instruction-tuning, and some calibration efforts. It introduces two complementary bias measures—a probabilistic BiasScore and an outcomes-based Relative Standard Deviation (RSD)—and demonstrates that these metrics can diverge under mitigation. A novel Leave-One-Out Calibration (LOOC) method is proposed to estimate and correct bias using only in-context demonstrations, achieving superior performance and bias reduction compared with existing calibration approaches while requiring less computation than LoRA fine-tuning. The findings emphasize that label bias remains a reliability barrier for LLMs and that robust, demonstrations-based calibration can meaningfully reduce bias, though residual biases persist and depend on task structure and demonstration selection. The work contributes a practical framework for measuring and mitigating bias in few-shot prompting, with implications for the reliability of LLM applications in diverse tasks.

Abstract

Large language models (LLMs) have shown remarkable adaptability to diverse tasks, by leveraging context prompts containing instructions, or minimal input-output examples. However, recent work revealed they also exhibit label bias -- an undesirable preference toward predicting certain answers over others. Still, detecting and measuring this bias reliably and at scale has remained relatively unexplored. In this study, we evaluate different approaches to quantifying label bias in a model's predictions, conducting a comprehensive investigation across 279 classification tasks and ten LLMs. Our investigation reveals substantial label bias in models both before and after debiasing attempts, as well as highlights the importance of outcomes-based evaluation metrics, which were not previously used in this regard. We further propose a novel label bias calibration method tailored for few-shot prompting, which outperforms recent calibration approaches for both improving performance and mitigating label bias. Our results emphasize that label bias in the predictions of LLMs remains a barrier to their reliability.

Figures (11)

• Figure 1: LLMs exhibit label bias---a tendency to output a given label regardless of the context and input (in this example, 'yes' over 'no'). In this work we evaluate LLM label bias across ten LLMs and 279 classification tasks, showing label bias is a major problem in LLMs.
• Figure 2: Performance (higher is better) and label bias metrics (lower is better) for Llama-2 pretrained and instruction-tuned models (7B/13B). Both performance and RSD improve with scale, instruction tuning, and number of demonstrations. In contrast, BiasScore is substantially worse after instruction tuning and does not improve when scaling models up in most evaluated settings.
• Figure 3: The effect of label bias mitigation methods on performance and bias for Llama-2 models. CC improves neither performance nor bias; DC and LoRA fine-tuning improve both; our Leave-One-Out Calibration (LOOC) method leads to the best performance among the calibration methods, and the overall lowest bias for $k\in\{8,16\}$.
• Figure 4: Label bias metrics for Llama-2 models (7B/13B), when evaluated on all tasks in our evaluation suite (All) vs. a subset of tasks with semantically equivalent labels (Sem. Eq. Labels). LLMs exhibit label bias even on tasks with semantically equivalent labels, such as multi-choice question answering.
• Figure 5: Label bias and performance metrics for Llama-2 (7B/13B) and Mistral (7B) models, when aggregated by the level of imbalance in the demonstrations set used for prompting the model, measured by the proportions of its most frequent label ($p_\uparrow$). For most tasks, label imbalance has only minor impact on both bias and performance, unless the imbalance is extreme. Instruction-tuned models are less sensitive to imbalance.