GRACE: Reinforcement Learning for Grounded Response and Abstention under Contextual Evidence

TL;DR

GRACE introduces a unified reinforcement-learning framework for ground- ing and abstention in retrieval-augmented generation. It combines a retriever-based data-construction pipeline that yields diverse evidence-sufficient and evidence-insufficient samples, a multi-stage gated reward that enforces explicit evidence grounding or transparent abstention, and a DAPO-inspired RL training regime for stable optimization. Empirical results on QASPER and HotpotQA show state-of-the-art overall accuracy and a favorable balance between accurate responses and refusals, with strong sample efficiency (2,000 annotations for a 4B model). The approach also maintains general capabilities and demonstrates robust cross-dataset generalization, while acknowledging limitations such as annotation dependence and computational cost. Overall, GRACE advances trustworthy RAG systems by tightly coupling grounding, abstention, and efficient learning in a single framework.

Abstract

Retrieval-Augmented Generation (RAG) integrates external knowledge to enhance Large Language Models (LLMs), yet systems remain susceptible to two critical flaws: providing correct answers without explicit grounded evidence and producing fabricated responses when the retrieved context is insufficient. While prior research has addressed these issues independently, a unified framework that integrates evidence-based grounding and reliable abstention is currently lacking. In this paper, we propose GRACE, a reinforcement-learning framework that simultaneously mitigates both types of flaws. GRACE employs a data construction method that utilizes heterogeneous retrievers to generate diverse training samples without manual annotation. A multi-stage gated reward function is then employed to train the model to assess evidence sufficiency, extract key supporting evidence, and provide answers or explicitly abstain. Experimental results on two benchmarks demonstrate that GRACE achieves state-of-the-art overall accuracy and strikes a favorable balance between accurate response and rejection, while requiring only 10% of the annotation costs of prior methods. Our code is available at https://github.com/YiboZhao624/Grace..

Figures (7)

• Figure 1: An example of responses generated by vanilla LLM and GRACE under different retrieval results.
• Figure 2: Overview of the full pipeline of our proposed method.
• Figure 3: Error analysis of the proposed method versus agentic RAG baselines. Classification accuracy is indicated by the "Correctly Classified" line. Proportions are calculated by the average results among different retrievers.
• Figure 4: Case study on Grace: evidence-path selection with Qwen3-4B. Green highlights the model’s key reasoning steps, while red marks incorrect attempts.
• Figure 5: Case study on Grace: evidence-path selection with Llama3.1-8B-Instruct. Green highlights the model’s key reasoning steps.