MR-Align: Meta-Reasoning Informed Factuality Alignment for Large Reasoning Models

TL;DR

This work addresses the persistent factuality gap in large reasoning models by shifting focus from output-level alignment to the quality of intermediate reasoning. It introduces MR-ALIGN, a meta-reasoning–informed framework that learns transition probabilities between atomic thinking states and uses a transition-aware reward to guide reasoning toward evidence-based, coherent conclusions. The approach combines a cognitive-grounded meta-reasoning annotation pipeline with Kahneman–Tversky Optimization, and estimates a latent meta-transition matrix via EM to produce a transition-modulated objective that improves accuracy and reduces misleading reasoning across five factual benchmarks, including long-form generation. The results demonstrate that aligning the reasoning process itself—rather than only the final answer—yields robust gains without external verifiers, suggesting a practical pathway to more trustworthy LRMs in factual domains.

Abstract

Large reasoning models (LRMs) show strong capabilities in complex reasoning, yet their marginal gains on evidence-dependent factual questions are limited. We find this limitation is partially attributable to a reasoning-answer hit gap, where the model identifies the correct facts during reasoning but fails to incorporate them into the final response, thereby reducing factual fidelity. To address this issue, we propose MR-ALIGN, a Meta-Reasoning informed alignment framework that enhances factuality without relying on external verifiers. MR-ALIGN quantifies state transition probabilities along the model's thinking process and constructs a transition-aware implicit reward that reinforces beneficial reasoning patterns while suppressing defective ones at the atomic thinking segments. This re-weighting reshapes token-level signals into probability-aware segment scores, encouraging coherent reasoning trajectories that are more conducive to factual correctness. Empirical evaluations across four factual QA datasets and one long-form factuality benchmark show that MR-ALIGN consistently improves accuracy and truthfulness while reducing misleading reasoning. These results highlight that aligning the reasoning process itself, rather than merely the outputs, is pivotal for advancing factuality in LRMs.

Figures (5)

• Figure 1: Illustration of Reasoning-Answer Hit Gap in Factual QA.
• Figure 2: MR-ALIGN adjusts reasoning transition for faithful response.
• Figure 3: Overview of MR-ALIGN Data Prepration Process.
• Figure 4: Meta-reasoning transition deltas for Qwen3-8B before vs. after MR-ALIGN.Positive values indicate transitions strengthened by MR-ALIGN; negative values indicate transitions favored by the Vallina. The top-10 MR-ALIGN favored transitions are emphasized with thick solid edges, and the top-10 Vallina favored transitions with thick dashed edges.
• Figure 5: Meta-reasoning transition advantages $w_i$ for the positive and negative subsets relative to the full training set. Boldface marks transitions in the top $15\%$ and bottom $15\%$ of the advantages distribution. .