Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Thinking, Faithful and Stable: Mitigating Hallucinations in LLMs

Chelsea Zou, Yiheng Yao, Basant Khalil

TL;DR

This work tackles hallucinations in LLMs during multi-step reasoning by shifting from outcome-only optimization to process-level supervision. It integrates two fine-grained uncertainty signals—token-level entropy spikes and self-confidence calibration—into a GRPO-style reinforcement learning objective to promote stable, faithful reasoning trajectories. On a math problem benchmark (MATH-500) with a Qwen3-0.6B base model and LoRA adapters, the approach yields gains in final accuracy and calibration (e.g., accuracy from 34% to 37%, ECE from 0.42 to 0.19) and reduces entropy fluctuations, while maintaining inference efficiency. These results demonstrate that explicit, continuous introspection signals can reduce hallucination-prone behavior and improve the trustworthiness of LLM reasoning, with practical implications for safer, more reliable AI-aided problem solving. Future work could extend to richer introspective signals and broader tasks to improve robustness and generalization.

Abstract

This project develops a self correcting framework for large language models (LLMs) that detects and mitigates hallucinations during multi-step reasoning. Rather than relying solely on final answer correctness, our approach leverages fine grained uncertainty signals: 1) self-assessed confidence alignment, and 2) token-level entropy spikes to detect unreliable and unfaithful reasoning in real time. We design a composite reward function that penalizes unjustified high confidence and entropy spikes, while encouraging stable and accurate reasoning trajectories. These signals guide a reinforcement learning (RL) policy that makes the model more introspective and shapes the model's generation behavior through confidence-aware reward feedback, improving not just outcome correctness but the coherence and faithfulness of their intermediate reasoning steps. Experiments show that our method improves both final answer accuracy and reasoning calibration, with ablations validating the individual contribution of each signal.

Thinking, Faithful and Stable: Mitigating Hallucinations in LLMs

TL;DR

This work tackles hallucinations in LLMs during multi-step reasoning by shifting from outcome-only optimization to process-level supervision. It integrates two fine-grained uncertainty signals—token-level entropy spikes and self-confidence calibration—into a GRPO-style reinforcement learning objective to promote stable, faithful reasoning trajectories. On a math problem benchmark (MATH-500) with a Qwen3-0.6B base model and LoRA adapters, the approach yields gains in final accuracy and calibration (e.g., accuracy from 34% to 37%, ECE from 0.42 to 0.19) and reduces entropy fluctuations, while maintaining inference efficiency. These results demonstrate that explicit, continuous introspection signals can reduce hallucination-prone behavior and improve the trustworthiness of LLM reasoning, with practical implications for safer, more reliable AI-aided problem solving. Future work could extend to richer introspective signals and broader tasks to improve robustness and generalization.

Abstract

This project develops a self correcting framework for large language models (LLMs) that detects and mitigates hallucinations during multi-step reasoning. Rather than relying solely on final answer correctness, our approach leverages fine grained uncertainty signals: 1) self-assessed confidence alignment, and 2) token-level entropy spikes to detect unreliable and unfaithful reasoning in real time. We design a composite reward function that penalizes unjustified high confidence and entropy spikes, while encouraging stable and accurate reasoning trajectories. These signals guide a reinforcement learning (RL) policy that makes the model more introspective and shapes the model's generation behavior through confidence-aware reward feedback, improving not just outcome correctness but the coherence and faithfulness of their intermediate reasoning steps. Experiments show that our method improves both final answer accuracy and reasoning calibration, with ablations validating the individual contribution of each signal.

Paper Structure

This paper contains 17 sections, 2 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Reward Function
  5. Experimental Setup
  6. GRPO Training Configuration
  7. Design Rationale
  8. Results
  9. Accuracy and Confidence Calibration
  10. Entropy Trends and Reasoning Stability
  11. Calibration Metrics
  12. Inference Performance
  13. Summary
  14. Training Dynamics
  15. Ablation Study
  16. ...and 2 more sections

Figures (1)

  • Figure 1: (a) The normalized (“z-scored”) entropy reward climbed steadily from –4 toward 0 over 100 GRPO steps, indicating improved stability. (b) The training loss decayed from $\sim2.0$ down toward $\sim0.3$, showing smooth convergence.