Context-aware Fairness Evaluation and Mitigation in LLMs

TL;DR

The paper tackles fairness in LLMs by addressing bias that accumulates across multi-turn conversations, a setting where training-time remedies are expensive and static inference methods often fail. It introduces Dynamic Neuron Masking, an inference-time, reversible framework that detects bias signals at the neuron level and gates them during generation using memory-aware cues. Key components include Behavioral Detection with turn-level scores $S_t$, Bias Neuron Identification via integrated gradients decomposed into local and carry components, a Memory Consistency Probe for $C_t$, and dynamic gating $g_l^{(t)} = \sigma(\alpha S_t + \beta C_t)$ to modulate neuron influence. On multilingual single-turn (PCT) and multi-turn (FairMT-Bench) benchmarks, the approach reduces bias while preserving fluency, faithfulness, and relevance, outperforming prompts, filters, steering, and static pruning. This work demonstrates practical, scalable fairness control for real-world conversational AI without retraining, highlighting the importance of memory-aware, context-sensitive interventions in dynamic dialogue settings.

Abstract

Large language models often display undesirable behaviors embedded in their internal representations, undermining fairness, inconsistency drift, amplification of harmful content, and the propagation of unwanted patterns during extended dialogue and conversations. Although training-time or data-centric methods attempt to reduce these effects, they are computationally expensive, irreversible once deployed, and slow to adapt to new conversational contexts. Pruning-based methods provide a flexible and transparent way to reduce bias by adjusting the neurons responsible for certain behaviors. However, most existing approaches are static; once a neuron is removed, the model loses the ability to adapt when the conversation or context changes. To address this, we propose a dynamic, reversible, pruning-based framework that detects context-aware neuron activations and applies adaptive masking to modulate their influence during generation. Our inference-time solution provides fine-grained, memory-aware mitigation with knowledge-preserved, more coherent behavior across multilingual single- and multi-turn dialogues, enabling dynamic fairness control in real-world conversational AI.

Figures (9)

• Figure 1: Comparison of inference-time mitigation methods. These methods are typically shallow, static, or single-turn oriented. Our framework uniquely combines adaptivity, multi-turn capability, reversibility, and fine-grained neuron-level control while preserving coherence and contextual integrity.
• Figure 2: Biases Build Up in Multi-Turn Conversations with LLMs: illustrates LLM can shift from aligned, cautious behaviour in single-turn prompts to producing harmful or stereotypical responses in multi-turn dialogues. Using tasks from the FairMT-Bench dataset, we show that even well-aligned models, while refusing to answer a biased question in isolation, begin to adopt and reinforce biases when exposed to subtle cues spread across multiple turns.
• Figure 3: The proposed framework mitigates biases in LLMs: Behavioral Analysis derives bias scores from model outputs; Bias Neuron Identification scores to neurons across dialogue context and history; Concept Testing probes memory consistency to measure bias persistence; and Adaptive Gating dynamically mask high-bias neurons while preserving coherence and context.
• Figure 4: Bias progression across dialogue turns and tasks. (Left) Turn-wise comparison of bias across different models shows an increase in later turns. (Middle) Bias across demographic groups highlights stronger signals for religion, race, and gender in multi-turn contexts.(Right) model–task interactions indicate task-specific vulnerability, with Scattered Questions (SQ) and Inference Misinformation (IM) producing higher bias scores across several models.
• Figure 5: Comparison of effective bias scores across dialogue turns, baseline (red, no masking) and mitigation (green, with masking) showing higher bias accumulation in the baseline (no masking) and substantial reduction with masking applied.