The Hallucination Tax of Reinforcement Finetuning

TL;DR

This paper identifies a hallucination tax in reinforcement finetuning (RFT), where LLMs become overconfident on unanswerable questions. It introduces Synthetic Unanswerable Math (SUM), a dataset of implicitly unanswerable math problems, to evaluate and train abstention behavior; SUM can be mixed into RFT at modest levels (around $10\%$) to substantially restore appropriate refusals with minimal loss to solvable-task accuracy. The authors show that SUM-trained models learn to reason about their own uncertainty and generalize abstention to out-of-domain tasks, improving trustworthiness and robustness across both math and factual QA benchmarks. These findings offer a practical path to calibrate RFT for safer deployment by leveraging inference-time computation to acknowledge knowledge boundaries. The work also discusses limitations and future directions for balancing reasoning capability with epistemic humility in diverse domains.

Abstract

Reinforcement finetuning (RFT) has become a standard approach for enhancing the reasoning capabilities of large language models (LLMs). However, its impact on model trustworthiness remains underexplored. In this work, we identify and systematically study a critical side effect of RFT, which we term the hallucination tax: a degradation in refusal behavior causing models to produce hallucinated answers to unanswerable questions confidently. To investigate this, we introduce SUM (Synthetic Unanswerable Math), a high-quality dataset of unanswerable math problems designed to probe models' ability to recognize an unanswerable question by reasoning from the insufficient or ambiguous information. Our results show that standard RFT training could reduce model refusal rates by more than 80%, which significantly increases model's tendency to hallucinate. We further demonstrate that incorporating just 10% SUM during RFT substantially restores appropriate refusal behavior, with minimal accuracy trade-offs on solvable tasks. Crucially, this approach enables LLMs to leverage inference-time compute to reason about their own uncertainty and knowledge boundaries, improving generalization not only to out-of-domain math problems but also to factual question answering tasks.

Figures (3)

• Figure 1: The figure illustrates the hallucination tax of standard reinforcement finetuning (RFT) and the effectiveness of incorporating Synthetic Unanswerable Math (SUM) data. Orange colored text indicates the deleted key information. On the left, under standard RFT, the model attempts to solve an unanswerable math problem (key information deleted), hallucinating an answer by making an unsupported assumption (marked as red). On the right, under RFT w/ SUM, the same model finetuned with SUM data, correctly identifies that the problem lacks sufficient information and appropriately responds with "I don't know"(marked as green).
• Figure 2: Refusal rate (higher is better) before and after RFT on three unanswerable datasets. The bar with backslashes denotes the performance before RFT; without backslashes denotes the performance after RFT. Different colors stand for different models. After RFT, the ability to refuse has a significant drop for all models.
• Figure 3: Learning dynamics of four LLMs during Reinforcement Finetuning (RFT) with varying mixing ratios (0%, 1%, 10%, 30%, and 50%) of unanswerable data. Each pair of plots shows the model's average performance over training steps: on unanswerable datasets (left column, reflecting refusal capability) and on answerable datasets (right column, reflecting accuracy on solvable tasks).