BARREL: Boundary-Aware Reasoning for Factual and Reliable LRMs

TL;DR

BARREL addresses the persistent reliability gap in Large Reasoning Models by identifying two overthinking patterns that yield overconfident, incorrect answers. It introduces a three-part pipeline—Knowledge Labeling, Reasoning Trace Construction for SFT, and GRPO Stage—to cultivate boundary-aware, concise reasoning and uncertainty-aware refusals. Empirical results show BARREL substantially improves factual reliability (e.g., from 39.33% to 61.48% Rel) on a DeepSeek-based 8B model while preserving accuracy, with medium-level rewards playing a key role. The work demonstrates a practical path toward more trustworthy System 2 LRMs that can admit ignorance when warranted and avoid unnecessary overconfidence.

Abstract

Recent advances in Large Reasoning Models (LRMs) have shown impressive capabilities in mathematical and logical reasoning. However, current LRMs rarely admit ignorance or respond with "I don't know". Instead, they often produce incorrect answers while showing undue confidence, raising concerns about their factual reliability. In this work, we identify two pathological reasoning patterns characterized by overthinking that contribute to the overconfident and incorrect answers: last-minute guessing and second-thought spiraling. To address these issues, we propose BARREL-a novel framework that promotes concise and boundary-aware factual reasoning. Our experiments show that BARREL-training increases the reliability of DeepSeek-R1-Distill-Llama-8B from 39.33% to 61.48%, while still achieving accuracy comparable to models finetuned on reasoning data generated by R1. These results demonstrate that our pilot study is inspiring to build more reliable and factual System 2 LRMs.

Figures (9)

• Figure 1: How reliable factual reasoning is expected to improve model performance. Left: Current LRMs rarely admit ignorance and often respond inconsistently. Right: Reliable LRMs should acknowledge unknowns and express known facts more consistently.
• Figure 2: Number of reasoning tokens used by LRMs when producing correct versus incorrect answers. We test on TruthfulQA across different types of reasoning models. More details at Appendix \ref{['sec:append_pilot']}.
• Figure 3: Left: The two current reasoning patterns of LRM: Last-minute Guessing, typically associated with unknown knowledge, and Second-thought Spiraling, which occurs despite known knowledge. Right: The BARREL pipeline addresses both cases by correcting overthinking tendencies and constructing SFT data accordingly, further enhanced with GRPO.
• Figure 4: Effect of the ratio of known data : unknown data on the factuality scores of SFT models.
• Figure 5: Effect of the reward on refusal on the factuality scores of GRPO models.