Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitation

TL;DR

High-stakes language tasks demand epistemic prudence; standard RLVR training rewards any answer and fails to incentivize abstention. Reinforced Hesitation (RH) introduces a ternary reward (+1 for correct, 0 for abstain, -λ for wrong) that encodes the cost of errors and yields a risk-aware boundary at $\lambda/(1+\lambda)$. Across Knights & Knaves puzzles, RH reveals a Pareto frontier of specialized models and enables two inference strategies—cascading and self-cascading—that convert hesitation into productive coordination, achieving high conditional accuracy with dramatically reduced verification. This work reframes abstention as a first-class objective, enabling trustworthy collaboration between AI systems and humans and offering scalable approaches to reduce compute while managing error costs in real-world deployments.

Abstract

Modern language models fail a fundamental requirement of trustworthy intelligence: knowing when not to answer. Despite achieving impressive accuracy on benchmarks, these models produce confident hallucinations, even when wrong answers carry catastrophic consequences. Our evaluations on GSM8K, MedQA and GPQA show frontier models almost never abstain despite explicit warnings of severe penalties, suggesting that prompts cannot override training that rewards any answer over no answer. As a remedy, we propose Reinforced Hesitation (RH): a modification to Reinforcement Learning from Verifiable Rewards (RLVR) to use ternary rewards (+1 correct, 0 abstention, -$λ$ error) instead of binary. Controlled experiments on logic puzzles reveal that varying $λ$ produces distinct models along a Pareto frontier, where each training penalty yields the optimal model for its corresponding risk regime: low penalties produce aggressive answerers, high penalties conservative abstainers. We then introduce two inference strategies that exploit trained abstention as a coordination signal: cascading routes queries through models with decreasing risk tolerance, while self-cascading re-queries the same model on abstention. Both outperform majority voting with lower computational cost. These results establish abstention as a first-class training objective that transforms ``I don't know'' from failure into a coordination signal, enabling models to earn trust through calibrated honesty about their limits.

Figures (15)

• Figure 1: Reinforced Hesitation creates a Pareto frontier of models trained under different penalties that reduce error rates through calibrated abstentions. This enables novel adaptive inference strategies.Right: Cross-penalty evaluation reveals mutual non-domination across our model family: each model achieves superior performance under specific evaluation penalties, with optimal models (orange) clustering near the diagonal where training and evaluation contexts align. This demonstrates that each training penalty produces a model that cannot be uniformly replaced by another. Top left: Cascading through models with decreasing risk aversion ($\lambda=10\rightarrow5\rightarrow2\rightarrow1\rightarrow0$) achieves efficient triage where each specialist handles problems matching its confidence regime. Bottom left: Models trained with different penalties form a Pareto frontier where higher $\lambda$ achieves lower error rates through calibrated abstentions, with no model dominating another across both accuracy and error rate dimensions.
• Figure 2: Penalty sensitivity of frontier models on GSM8K.Left: Expected reward $r(\lambda)=p(\text{correct})-\lambda\,p(\text{wrong})$ for $\lambda\in\{1,5,25,100\}$; red dashed line marks $r=0$ baseline. Middle: Frontier models rarely choose to abstain, even when faced with penalties of magnitude 100. Right: Despite the high penalty values, the rate of wrong answers remains high across various models.
• Figure 3: GSM8K prompt modification. We augment the standard prompt with explicit abstention instructions and reward structure.
• Figure 4: An example Knights & Knaves puzzle. See Appendix \ref{['sec:appendix_training_prompts']} for the complete training prompt.
• Figure 5: Training dynamics across penalty values.Left: Mean training reward trajectories diverge by penalty, with $\lambda=0$ achieving highest reward while $\lambda=10$ shows dramatic mid-training dip and recovery. Middle: Validation rewards closely track training patterns, confirming generalization across all penalty regimes. Right: Response length decreases with higher penalties, compressing from 3000+ tokens to 1200-2200 tokens as models learn concise uncertainty expression.