CRAFT: Calibrated Reasoning with Answer-Faithful Traces via Reinforcement Learning for Multi-Hop Question Answering

TL;DR

CRAFT tackles the core challenges of reliable, faithful reasoning in retrieval-augmented multi-hop QA by introducing a GRPO-based reinforcement learning framework that trains models to generate structured, machine-auditable reasoning traces. It couples deterministic rewards for format, citation, and answer correctness with judge-based rewards assessing faithfulness, enabling a controllable trace space to study how structure and scale affect performance. Across three benchmarks, CRAFT shows consistent gains in both accuracy and faithfulness across model scales, with its 7B open-source model matching or exceeding several closed-source API models under multiple trace configurations. The work highlights the importance of structured traces and judge-driven supervision for dependable reasoning, and provides detailed analyses of training dynamics and trace quality to guide future design choices. Overall, CRAFT advances practical, explainable QA by ensuring that generated reasoning is both correct and grounded in retrieved evidence, enabling safer and more trustworthy AI-assisted information synthesis.

Abstract

Retrieval-augmented generation (RAG) is widely used to ground Large Language Models (LLMs) for multi-hop question answering. Recent work mainly focused on improving answer accuracy via fine-tuning and structured or reinforcement-based optimization. However, reliable reasoning in response generation faces three challenges: 1) Reasoning Collapse. Reasoning in multi-hop QA is inherently complex due to multi-hop composition and is further destabilized by noisy retrieval. 2) Reasoning-answer inconsistency. Due to the intrinsic uncertainty of LLM generation and exposure to evidence--distractor mixtures, models may produce correct answers that are not faithfully supported by their intermediate reasoning or evidence. 3) Loss of format control. Traditional chain-of-thought generation often deviates from required structured output formats, leading to incomplete or malformed structured content. To address these challenges, we propose CRAFT (Calibrated Reasoning with Answer-Faithful Traces), a Group Relative Policy Optimization (GRPO) based reinforcement learning framework that trains models to perform faithful reasoning during response generation. CRAFT employs dual reward mechanisms to optimize multi-hop reasoning: deterministic rewards ensure structural correctness while judge-based rewards verify semantic faithfulness. This optimization framework supports controllable trace variants that enable systematic analysis of how structure and scale affect reasoning performance and faithfulness. Experiments on three multi-hop QA benchmarks show that CRAFT improves both answer accuracy and reasoning faithfulness across model scales, with the CRAFT 7B model achieving competitive performance with closed-source LLMs across multiple reasoning trace settings.

Figures (8)

• Figure 1: Challenges in Multi-hop QA Tasks.
• Figure 2: Overview of CRAFT. The framework comprises three components: CRAFT Interface (top) defines structured trace templates (CRAFT$_{\text{v1}}$--CRAFT$_{\text{v5}}$); Decomposed Reward Auditor (center) provides four reward signals ($R_{\mathrm{fmt}}$, $R_{\mathrm{gold}}$, $R_{\mathrm{faith}}$, $R_{\mathrm{ans}}$); Faithfulness Audit Principle (right) enables machine-checkable reasoning verification.
• Figure 3: Training dynamics of GRPO models across different scales (0.5B, 1.5B, 3B, 7B) using CRAFT$_{\text{v1}}$ template. The left y-axis shows Total Reward, while the right y-axis displays sub-rewards: $R_{\mathrm{fmt}}$, $R_{\mathrm{gold}}$, $R_{\mathrm{faith}}$, and $R_{\mathrm{ans}}$. All models demonstrate smooth convergence with larger models achieving higher total rewards, validating the effectiveness of our multi-reward GRPO framework.
• Figure 4: Training data distribution visualized using t-SNE. The dataset consists of 20,000 samples: 10,000 MuSiQue (black), 5,000 HotpotQA (red), and 5,000 2WikiMHQA (gray). The clustering patterns indicate distinct linguistic characteristics across different multi-hop QA datasets.
• Figure 5: Training dynamics using CRAFT$_{\text{v2}}$ template.