Making Bias Non-Predictive: Training Robust LLM Judges via Reinforcement Learning

TL;DR

This work tackles the vulnerability of LLM-based judges to cognitive biases such as bandwagon and authority cues. It introduces Epistemic Independence Training (EIT), a reinforcement learning framework that makes bias cues non-predictive of reward through a balanced bias-conflict setup and a bias-penalizing reward, optimized with GRPO. Empirical results on Qwen3-1.7B and Qwen3-4B with MMLU-Pro show that EIT improves both accuracy and robustness to in-domain and out-of-domain biases, often outperforming larger untrained models and exhibiting transferable robustness to unseen biases. Qualitative analyses reveal that EIT promotes substantive domain engagement, explicit verification, and reasoned disagreement, reducing performative independence observed in SFT. These findings advance reliable LLM evaluation by reducing susceptibility to social cues and enabling more faithful reasoning under bias pressure, with potential extensions to more open-ended tasks and broader bias categories.

Abstract

Large language models (LLMs) increasingly serve as automated judges, yet they remain susceptible to cognitive biases -- often altering their reasoning when faced with spurious prompt-level cues such as consensus claims or authority appeals. Existing mitigations via prompting or supervised fine-tuning fail to generalize, as they modify surface behavior without changing the optimization objective that makes bias cues predictive. To address this gap, we propose Epistemic Independence Training (EIT), a reinforcement learning framework grounded in a key principle: to learn independence, bias cues must be made non-predictive of reward. EIT operationalizes this through a balanced conflict strategy where bias signals are equally likely to support correct and incorrect answers, combined with a reward design that penalizes bias-following without rewarding bias agreement. Experiments on Qwen3-4B demonstrate that EIT improves both accuracy and robustness under adversarial biases, while preserving performance when bias aligns with truth. Notably, models trained only on bandwagon bias generalize to unseen bias types such as authority and distraction, indicating that EIT induces transferable epistemic independence rather than bias-specific heuristics. Code and data are available at https://anonymous.4open.science/r/bias-mitigation-with-rl-BC47.

Figures (11)

• Figure 1: The fragility of LLM judgment under bandwagon bias. Left: In a clean setting, OpenAI-o1 correctly identifies that the Great Wall of China is not visible from space with the naked eye. Right: When exposed to bandwagon bias (a fabricated consensus claiming visibility), the same model succumbs to social pressure and reverses its correct judgment.
• Figure 2: Overview of EIT. Training phase (left): Questions are injected with bandwagon bias using the conflict strategy---correct-bias (green) points to the right answer, wrong-bias (red) points to the wrong answer. The policy $\pi_\theta$ generates multiple responses evaluated by our hierarchical reward: $\mathcal{R}_{\text{acc}}$ (accuracy), $\mathcal{R}_{\text{struct}}$ (format), and $\mathcal{R}_{\text{ind}}$ (independence with asymmetric incentives). Test phase (right): The trained model $\pi^*$ is evaluated on in-domain bandwagon bias and three out-of-domain biases (authority, position, distraction) to test whether epistemic independence generalizes beyond the training distribution.
• Figure 3: EIT training dynamics. Both models show stable convergence with reward plateauing after the marked checkpoints.
• Figure 4: Comparison of Qwen3-4B+EIT against larger Qwen3 models under bias conditions on the MMLU-Pro test set. Despite having fewer parameters, EIT-trained Qwen3-4B outperforms both Qwen3-8B and Qwen3-14B under wrong-bias settings, demonstrating that targeted training is more effective than model scaling for bias robustness.
• Figure 5: Impact of Training Data Composition. Comparison of our balanced Conflict strategy (50/50 correct/wrong bias) versus a Wrong-Only strategy (training solely on samples where bias predicts the wrong answer). While Wrong-Only artificially boosts performance when the bias is misleading (Wrong-Bias), it causes a catastrophic drop when the bias happens to be correct (Correct-Bias), indicating the model learned a simple "reverse-bias" shortcut. In contrast, EIT's Conflict strategy maintains high robustness across both scenarios.