Identifying Features Associated with Bias Against 93 Stigmatized Groups in Language Models and Guardrail Model Safety Mitigation

TL;DR

This study investigates bias against stigmatized groups in open-source LLMs by linking bias to six psychology-informed stigma features and stigma cluster types, across 93 stigmas using the SocialStigmaQA benchmark. It introduces classification-based prompts to elicit human-LLM feature ratings, quantifies bias with extensive prompt variants, and evaluates guardrail-based mitigation. Results show weak-to-moderate alignment between human and LLM feature ratings, substantial bias driven by stigma type and prompt style (with 60% bias for Threatening stigmas and 11% for Sociodemographic), and partial reduction of bias via guardrails that often fail to identify intent behind biased prompts. The findings suggest guardrails can help but are not a complete solution, underscoring the need for improved intent-aware moderation and stigma-informed evaluation to responsibly deploy LLMs around stigmatized groups.

Abstract

Large language models (LLMs) have been shown to exhibit social bias, however, bias towards non-protected stigmatized identities remain understudied. Furthermore, what social features of stigmas are associated with bias in LLM outputs is unknown. From psychology literature, it has been shown that stigmas contain six shared social features: aesthetics, concealability, course, disruptiveness, origin, and peril. In this study, we investigate if human and LLM ratings of the features of stigmas, along with prompt style and type of stigma, have effect on bias towards stigmatized groups in LLM outputs. We measure bias against 93 stigmatized groups across three widely used LLMs (Granite 3.0-8B, Llama-3.1-8B, Mistral-7B) using SocialStigmaQA, a benchmark that includes 37 social scenarios about stigmatized identities; for example deciding wether to recommend them for an internship. We find that stigmas rated by humans to be highly perilous (e.g., being a gang member or having HIV) have the most biased outputs from SocialStigmaQA prompts (60% of outputs from all models) while sociodemographic stigmas (e.g. Asian-American or old age) have the least amount of biased outputs (11%). We test if the amount of biased outputs could be decreased by using guardrail models, models meant to identify harmful input, using each LLM's respective guardrail model (Granite Guardian 3.0, Llama Guard 3.0, Mistral Moderation API). We find that bias decreases significantly by 10.4%, 1.4%, and 7.8%, respectively. However, we show that features with significant effect on bias remain unchanged post-mitigation and that guardrail models often fail to recognize the intent of bias in prompts. This work has implications for using LLMs in scenarios involving stigmatized groups and we suggest future work towards improving guardrail models for bias mitigation.

Figures (4)

• Figure 1: Comparison between humans and the three language models in rating each stigma by six features. Models rate stigmas, on average, to be more perilous, disruptive, and visible than determined by humans.
• Figure 2: Correlation of LLM to humans for ratings of each feature dimension. LLMs rate stigmas that are not very visible to be highly visible, and vice versa.
• Figure 3: Change in bias per stigma cluster type. Before mitigation is each guardrail model's respective language model (e.g. Granite Guardian's before mitigation is results from the Granite LLM).
• Figure 4: Change in bias per prompt style. Before mitigation is each guardrail model's respective language model (e.g. Granite Guardian's before mitigation is results from the Granite LLM)