Assessing Bias in Metric Models for LLM Open-Ended Generation Bias Benchmarks
Nathaniel Demchak, Xin Guan, Zekun Wu, Ziyi Xu, Adriano Koshiyama, Emre Kazim
TL;DR
Open-generation bias benchmarks rely on classifiers that may inherit sociocultural biases, risking biased conclusions about LLM outputs. The study uses counterfactuals on the MGSD dataset to systematically perturb stereotype-related prefixes and applies SHAP to identify which demographic cues drive predictions, evaluating Detoxify, RegardV3, DistilBERTSentiment, and Vader. Key findings show substantial disparities across race, religion, and gender descriptors, with RegardV3 being most biased and certain groups eliciting stronger negative scores; SHAP analysis confirms the impact of specific descriptors. The work highlights a need for robust bias metric models and complementary explainability to ensure fair and reliable bias assessment in open-ended generation.
Abstract
Open-generation bias benchmarks evaluate social biases in Large Language Models (LLMs) by analyzing their outputs. However, the classifiers used in analysis often have inherent biases, leading to unfair conclusions. This study examines such biases in open-generation benchmarks like BOLD and SAGED. Using the MGSD dataset, we conduct two experiments. The first uses counterfactuals to measure prediction variations across demographic groups by altering stereotype-related prefixes. The second applies explainability tools (SHAP) to validate that the observed biases stem from these counterfactuals. Results reveal unequal treatment of demographic descriptors, calling for more robust bias metric models.