Train for Truth, Keep the Skills: Binary Retrieval-Augmented Reward Mitigates Hallucinations

TL;DR

This work introduces Binary Retrieval-Augmented Reward (Binary RAR), an online reinforcement learning method that uses a binary, retrieval-verified factuality signal to mitigate extrinsic hallucinations in large language models. By verifying outputs against retrieved evidence and rewarding only fully consistent responses, Binary RAR improves factual accuracy in long-form generation and enables calibrated abstention in short-form QA, all while preserving instruction-following, knowledge retention, reasoning, and coding abilities. The approach avoids reward hacking typical of continuous rewards and demonstrates a favorable hallucination–utility trade-off across multiple model sizes and benchmarks. These results suggest that simple binary signals, coupled with retrieval-augmented verification, offer a robust and scalable path toward safer, more reliable language models with minimal performance penalties. The method leverages online RL with GRPO, a retrieval-and-verification pipeline, and strategic dataset curation to achieve consistent gains in factual reliability and controlled abstention.

Abstract

Language models often generate factually incorrect information unsupported by their training data, a phenomenon known as extrinsic hallucination. Existing mitigation approaches often degrade performance on open-ended generation and downstream tasks, limiting their practical utility. We propose an online reinforcement learning method using a novel binary retrieval-augmented reward (RAR) to address this tradeoff. Unlike continuous reward schemes, our approach assigns a reward of one only when the model's output is entirely factually correct, and zero otherwise. We evaluate our method on Qwen3 reasoning models across diverse tasks. For open-ended generation, binary RAR achieves a 39.3% reduction in hallucination rates, substantially outperforming both supervised training and continuous-reward RL baselines. In short-form question answering, the model learns calibrated abstention, strategically outputting "I don't know" when faced with insufficient parametric knowledge. This yields 44.4% and 21.7% fewer incorrect answers on PopQA and GPQA, respectively. Crucially, these factuality gains come without performance degradation on instruction following, math, or code, whereas continuous-reward RL, despite improving factuality, induces quality regressions.

Figures (12)

• Figure 1: Overview of Binary Retrieval-Augmented Reward (Binary RAR). Left: Reinforcement learning with Binary RAR assigns a binary reward based on retrieval-verified factual correctness. Right: Binary RAR achieves the best hallucination–utility tradeoff among all post-training baselines.
• Figure 2: Informativeness in long-form generation. Left: On Biography, Binary RAR cuts the total number of claims but keeps correct claims nearly the same, showing selective filtering of uncertain content. Right: On AlpacaEval, Binary RAR gives shorter answers with similar win rates, showing it stays concise without losing quality.
• Figure 3: Abstention behavior in short-form question answering. Binary RAR leads the model to abstain on uncertain questions rather than producing incorrect answers, preserving accuracy for attempted ones.
• Figure 4: Hallucination–utility tradeoff scatter plot for ablations on different KL coefficients (left) and reward designs (right).
• Figure 5: Comparing Binary RAR, VeriScore, and LM Judge on LM responses. Each reward provides a score and reason, with green and red highlights marking correct and incorrect facts; Binary RAR penalizes factual errors most reliably.