Quantifying Fairness in LLMs Beyond Tokens: A Semantic and Statistical Perspective

TL;DR

FiSCo introduces a principled framework for evaluating fairness in long-form LLM outputs by moving beyond token-level or sentiment analysis to fine-grained claim-level semantics. It formalizes group counterfactual fairness, decomposes responses into semantic claims, and uses bidirectional entailment to compute a robust similarity metric. Inter- and intra-group similarities are statistically compared with Welch's t-test to detect significant biases while mitigating stochastic output variability. The approach is validated on a large, human-annotated long-form fairness dataset and demonstrates superior sensitivity to nuanced biases across gender, race, and age, offering a scalable tool for diagnosing and mitigating bias in real-world LLM deployments.

Abstract

Large Language Models (LLMs) often generate responses with inherent biases, undermining their reliability in real-world applications. Existing evaluation methods often overlook biases in long-form responses and the intrinsic variability of LLM outputs. To address these challenges, we propose FiSCo (Fine-grained Semantic Comparison), a novel statistical framework to evaluate group-level fairness in LLMs by detecting subtle semantic differences in long-form responses across demographic groups. Unlike prior work focusing on sentiment or token-level comparisons, FiSCo goes beyond surface-level analysis by operating at the claim level, leveraging entailment checks to assess the consistency of meaning across responses. We decompose model outputs into semantically distinct claims and apply statistical hypothesis testing to compare inter- and intra-group similarities, enabling robust detection of subtle biases. We formalize a new group counterfactual fairness definition and validate FiSCo on both synthetic and human-annotated datasets spanning gender, race, and age. Experiments show that FiSCo more reliably identifies nuanced biases while reducing the impact of stochastic LLM variability, outperforming various evaluation metrics.

Figures (10)

• Figure 1: By comparing responses to identical prompts that differ only in the personas’ genders, repeatedly, FiSCo can detect subtle differences (shown in bold) that reveal potential gender bias with statistical significance. In above example, Jack will be replaced by a group of other male names while Jackie will be replaced by a group of female names.
• Figure 2: Overview of FiSCo pipeline for evaluating group-level fairness. First, prompts are adapted for each demographic group (e.g., male vs. female), and responses are generated via LLMs. Each response is decomposed into semantic claims, and entailment relationships are computed across response pairs. A fine-grained similarity score is calculated between each response pair. Finally, Welch’s t-test compares inter-group and intra-group similarity distributions to assess the statistical significance of bias.
• Figure 2: Performance comparison of similarity measurements on synthetic and human-annotated datasets. The best and second-best scores for each dataset are shown in bold and underlined, respectively. The confidence interval is approximated by bootstrapping. $^{+}$ indicates a p-value below 0.01 and * indicates a p-value below 0.05.
• Figure 3: The t-SNE plot of different similarity metrics on the gender bias dataset.
• Figure 4: Annotation UI for suitability evaluation.