Learning to Reason Faithfully through Step-Level Faithfulness Maximization

TL;DR

FaithRL tackles the limitation of sparse outcome rewards in RL for LLM reasoning by formalizing a reasoning faithfulness objective. It combines a geometric Truthful Helpful Score based reward with a step level faithfulness verifier to credit only steps that are strictly supported by evidence, achieving reduced hallucinations and improved correctness across multi hop QA tasks. The method demonstrates strong in domain and out of distribution performance gains and shows that faithfulness driven learning yields robust and transferable reasoning patterns. While adding modest computational overhead, FaithRL provides a principled path to reliable and verifiable reasoning in complex AI systems.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has markedly improved the performance of Large Language Models (LLMs) on tasks requiring multi-step reasoning. However, most RLVR pipelines rely on sparse outcome-based rewards, providing little supervision over intermediate steps and thus encouraging over-confidence and spurious reasoning, which in turn increases hallucinations. To address this, we propose FaithRL, a general reinforcement learning framework that directly optimizes reasoning faithfulness. We formalize a faithfulness-maximization objective and theoretically show that optimizing it mitigates over-confidence. To instantiate this objective, we introduce a geometric reward design and a faithfulness-aware advantage modulation mechanism that assigns step-level credit by penalizing unsupported steps while preserving valid partial derivations. Across diverse backbones and benchmarks, FaithRL consistently reduces hallucination rates while maintaining (and often improving) answer correctness. Further analysis confirms that FaithRL increases step-wise reasoning faithfulness and generalizes robustly. Our code is available at https://github.com/aintdoin/FaithRL.

Figures (8)

• Figure 1: Comparison between factuality-driven methods and FaithRL. Unlike factuality-driven methods that induce hallucinations by rewarding correct but unfaithful steps, FaithRL penalizes such irrelevance to ensure a faithful and correct answer.
• Figure 2: Overall framework of FaithRL. Taking a query and knowledge $\mathcal{K}$, the model first uses offline validation to set baseline rates ($x_0, y_0$). During online group rollout, these rates form a geometric reward based on answer correctness. Simultaneously, the verifier $\mathcal{V}$ checks alignment with evidence $\mathcal{E}$ to derive token-level advantage modulation for the policy update.
• Figure 3: Truthful Helpfulness Score (THS).
• Figure 4: Comparison of the faithful step ratio among FaithRL, TruthRL, and GRPO during training.
• Figure 5: Model performance on OOD tasks. (a) illustrates the outcome correctness distribution across datasets. (b) displays the faithfulness of the reasoning process. For each dataset, the left bar in each pair represents the GRPO baseline, and the right bar represents FaithRL. Faithful (Correct) denotes reasoning steps that are verified as faithful within correctly answered samples.

Theorems & Definitions (8)

• Corollary 3.2: Answerability Partition
• Theorem 4.1: Asymptotic stability of refusal strategy
• Theorem 4.2: Geometric alignment with THS
• Theorem 4.3: Optimization Consistency and Bias Rectification
• proof
• proof
• proof
• proof