Characterizing Selective Refusal Bias in Large Language Models

TL;DR

Safety guardrails in large language models aim to prevent harm but may introduce selective refusal bias across demographic groups. The authors build a framework using template prompts, the WildGuardMix dataset, and multiple models to quantify refusal rates, refusal lengths, and response types for individual and intersectional identities. They find systematic bias with higher refusals for marginalized groups and model-specific patterns, plus an indirect attack demonstrating guardrail vulnerabilities. The work highlights the need for more equitable and robust guardrails that evaluate requests by content and intent rather than demographic identifiers, and suggests directions for strengthening protection against indirect exploitation.

Abstract

Safety guardrails in large language models(LLMs) are developed to prevent malicious users from generating toxic content at a large scale. However, these measures can inadvertently introduce or reflect new biases, as LLMs may refuse to generate harmful content targeting some demographic groups and not others. We explore this selective refusal bias in LLM guardrails through the lens of refusal rates of targeted individual and intersectional demographic groups, types of LLM responses, and length of generated refusals. Our results show evidence of selective refusal bias across gender, sexual orientation, nationality, and religion attributes. This leads us to investigate additional safety implications via an indirect attack, where we target previously refused groups. Our findings emphasize the need for more equitable and robust performance in safety guardrails across demographic groups.

Figures (12)

• Figure 1: Example of selective bias refusal. The two toxic prompts only differ in the subject's demographic group (Mexican versus French), but the model only refuses to answer the input regarding Mexican people.
• Figure 2: Response rates across gender, sexual orientation, and religion attributes. Results are averaged across all models in our study. Individual results for each model are in Figure \ref{['fig:refusal_rates_all_models']} in the Appendix.
• Figure 3: Nationality refusal patterns averaged over all investigated LLMs. Individual results for each model are in Figure \ref{['fig:refusal_rates_nationality']} in the Appendix.
• Figure 4: Refusal response rates for intersectional groups (blue bars) and their respective individual groups (pink and black lines) across intersectional group settings. Results are averaged across GPT-4o and Llama-70b models. Individual results for each model are in Figures \ref{['fig:intesectional_refusal_rate_llama']} and \ref{['fig:intesectional_refusal_rate_gpt']} in the appendix.
• Figure 5: Average length of refusal responses for all models across demographic groups. Refusal length for each attribute-model pair as well as the total number of prompts used for each pair are provided in Tables \ref{['tab:refusal_length_attmodel']} and \ref{['tab:Total_N_prompts']} in the appendix.