Reinforced Informativeness Optimization for Long-Form Retrieval-Augmented Generation

TL;DR

This paper tackles long-form retrieval-augmented generation (LFQA) by introducing RioRAG, an RL framework that directly optimizes informativeness for extended answers without requiring scarce supervised data. It couples reinforced informativeness optimization with a nugget-centric hierarchical reward model, enabling precise, cross-document factual coverage through three stages: nugget extraction, cross-webpage checklist integration, and generation evaluation, plus a length-decay mechanism to control verbosity. Empirical results on LongFact and RAGChecker demonstrate that RioRAG outperforms prompt-based, supervised fine-tuned, and offline RL baselines, with ablations confirming the value of each component. The work advances practical LFQA by promoting fact-centric evaluation and on-policy learning, with implications for scalable, trustworthy long-form Q&A in knowledge-intensive domains; future work includes multilingual extensions and human-in-the-loop reward refinement.

Abstract

Long-form question answering (LFQA) presents unique challenges for large language models, requiring the synthesis of coherent, paragraph-length answers. While retrieval-augmented generation (RAG) systems have emerged as a promising solution, existing research struggles with key limitations: the scarcity of high-quality training data for long-form generation, the compounding risk of hallucination in extended outputs, and the absence of reliable evaluation metrics for factual completeness. In this paper, we propose RioRAG, a novel reinforcement learning (RL) framework that advances long-form RAG through reinforced informativeness optimization. Our approach introduces two fundamental innovations to address the core challenges. First, we develop an RL training paradigm of reinforced informativeness optimization that directly optimizes informativeness and effectively addresses the slow-thinking deficit in conventional RAG systems, bypassing the need for expensive supervised data. Second, we propose a nugget-centric hierarchical reward modeling approach that enables precise assessment of long-form answers through a three-stage process: extracting the nugget from every source webpage, constructing a nugget claim checklist, and computing rewards based on factual alignment. Extensive experiments on two LFQA benchmarks LongFact and RAGChecker demonstrate the effectiveness of the proposed method. Our codes are available at https://github.com/RUCAIBox/RioRAG.

Figures (3)

• Figure 1: Overall illustration of the proposed RL-based RioRAG framework.
• Figure 2: In-depth analysis on scaling law and RL cold-start.
• Figure 3: Analysis on co-evolution of generation length and reward during RL training.