Decomposition-Enhanced Training for Post-Hoc Attributions In Language Models

TL;DR

DecompTune reframes post-hoc attribution as a reasoning task by training language models to decompose answers into context-grounded units before generating attributions. It uses a two-stage pipeline (SFT followed by GRPO) and a decomposition-annotated dataset (~90k samples) to teach models how to ground complex, multi-hop, and abstractive answers. Across diverse benchmarks, DecompTune outperforms existing post-hoc attribution methods and matches or exceeds frontier models, especially for multi-hop and abstractive QA. The work advances grounding and interpretability in open-source LMs with practical implications for trustworthy AI in long-document QA.

Abstract

Large language models (LLMs) are increasingly used for long-document question answering, where reliable attribution to sources is critical for trust. Existing post-hoc attribution methods work well for extractive QA but struggle in multi-hop, abstractive, and semi-extractive settings, where answers synthesize information across passages. To address these challenges, we argue that post-hoc attribution can be reframed as a reasoning problem, where answers are decomposed into constituent units, each tied to specific context. We first show that prompting models to generate such decompositions alongside attributions improves performance. Building on this, we introduce DecompTune, a post-training method that teaches models to produce answer decompositions as intermediate reasoning steps. We curate a diverse dataset of complex QA tasks, annotated with decompositions by a strong LLM, and post-train Qwen-2.5 (7B and 14B) using a two-stage SFT + GRPO pipeline with task-specific curated rewards. Across extensive experiments and ablations, DecompTune substantially improves attribution quality, outperforming prior methods and matching or exceeding state-of-the-art frontier models.

Figures (4)

• Figure 1: Overview of our method DecompTune. We train small- to medium-sized language models to incorporate answer decomposition as an intermediate reasoning step prior to generating final attributions. To enable this, we leverage decomposition traces produced by an annotator LLM and adopt a two-stage training strategy: SFT to instill decomposition-aware reasoning, followed by GRPO with task-specific continuous and verifiable rewards to further refine attribution quality.
• Figure 2: Decomposition as intermediate step helps in attribution. Prompting LLMs to decompose an answer as an intermediate step, then attribute, performs better than solely prompting them to only generate attributions, even with chain-of-thought prompting.
• Figure 3: Effect of Context-Length on Post-Hoc Attribution. IS and CS score on samples from all datasets of different lengths, grouped into 10 equal quantile bins. DecompTune (7B) outperforms GPT 4.1 mini on IS, and is closely matched on CS. When coupled with BM25 based search to reduce context, all models gain significantly on IS, but CS for GPT-4.1 drops while DecompTune improves.
• Figure 4: Validity reward ensures that the output citations are present verbatim in the context.