GenProve: Learning to Generate Text with Fine-Grained Provenance

TL;DR

This work tackles the problem of unreliable, unverifiable generation by proposing generation-time fine-grained provenance, where each generated sentence is accompanied by sentence-level evidence links and explicit relation types (Quotation, Compression, Inference). It introduces ReFInE, an expert-annotated dataset, and GenProve, a two-stage training framework that first uses supervised fine-tuning and then GRPO-based reinforcement learning with a composite reward balancing content fidelity and provenance correctness. Empirical results show GenProve achieves state-of-the-art performance across 14 LLMs on both answer quality and provenance accuracy, with strong alignment between automatic judge scores and human evaluations. A key finding is that while models excel at direct quotation, they struggle with inference-based provenance, pointing to a frontier in verifiable reasoning for multi-document generation.

Abstract

Large language models (LLM) often hallucinate, and while adding citations is a common solution, it is frequently insufficient for accountability as users struggle to verify how a cited source supports a generated claim. Existing methods are typically coarse-grained and fail to distinguish between direct quotes and complex reasoning. In this paper, we introduce Generation-time Fine-grained Provenance, a task where models must generate fluent answers while simultaneously producing structured, sentence-level provenance triples. To enable this, we present ReFInE (Relation-aware Fine-grained Interpretability & Evidence), a dataset featuring expert verified annotations that distinguish between Quotation, Compression, and Inference. Building on ReFInE, we propose GenProve, a framework that combines Supervised Fine-Tuning (SFT) with Group Relative Policy Optimization (GRPO). By optimizing a composite reward for answer fidelity and provenance correctness, GenProve significantly outperforms 14 strong LLMs in joint evaluation. Crucially, our analysis uncovers a reasoning gap where models excel at surface-level quotation but struggle significantly with inference-based provenance, suggesting that verifiable reasoning remains a frontier challenge distinct from surface-level citation.

Figures (18)

• Figure 1: Overview of Generation-time Fine-grained Provenance. Given a query and source documents, the model simultaneously produces the answer and structured triples (DocID, SentID, Relation) to explain how the evidence supports each generated sentence.
• Figure 2: The construction pipeline of ReFInE. The process ensures high-quality provenance supervision through three stages: (1) preprocessing, (2) LLM-assisted annotation with filtering, and (3) reconstruction with rigorous human-in-the-loop expert validation to verify evidence sufficiency and relation correctness.
• Figure 3: Relation-type distribution in ReFInE.
• Figure 4: The GenProve framework. The model first undergoes SFT for instruction following and format learning. It is then aligned using GRPO with a composite reward mechanism that jointly optimizes for answer fidelity (content similarity reward) and fine-grained provenance accuracy (F1 Reward).
• Figure 5: Performance breakdown by relation type (F1 score). The heatmap reveals a reasoning gap: while most models handle verbatim Quotation well, they struggle significantly with Inference. GenProve consistently outperforms baselines, showing the largest gains in complex provenance tasks (Compression and Inference).