Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Transparent and Robust RAG: Adaptive-Reward Reinforcement Learning for Decision Traceability

Jingyi Ren, Yekun Xu, Xiaolong Wang, Weitao Li, Weizhi Ma, Yang Liu

TL;DR

ARENA introduces a transparent and robust RL-based RAG framework that couples a structured evidence–reasoning–answer protocol with KL-divergence stabilization using the $k_2$ estimator and adaptive, task-specific rewards. The approach enforces auditable grounding by requiring explicit <relevance> indices and reasoning traces, while stabilizing policy optimization via GRPO with the stabilized KL term. Four reward components—format, accuracy, relevance, and bonus—drive well-formed, evidence-grounded outputs, achieving 10–30% gains on three multi-hop QA benchmarks and generalizing across models and retrieval settings. The work demonstrates that smaller open-source backbones can approach or match some closed-source models in multi-hop reasoning when trained with ARENA, and provides extensive ablation and cross-task analyses along with public code release.

Abstract

Retrieval-Augmented Generation (RAG) delivers substantial value in knowledge-intensive applications. Many recent works use reinforcement learning (RL) to elicit strong reasoning in RAG generators. However, two key challenges remain unresolved: (1) Transparency: most prior methods do not explicitly indicate which references are actually used during the reasoning that leads to the final answer, limiting interpretability and visibility; (2) Stability: the KL divergence estimator used in existing RL-based approaches may cause gradient spikes, leading to unstable training. To address these challenges, we propose Adaptive-Rewarded Evidence Navigation Agent (ARENA), a transparent and robust RAG generator framework trained via RL with designed rewards. Based on our structured protocol, KL divergence stabilization, and adaptive reward calculation modules, ARENA enables the RAG generator to identify key evidence, perform structured reasoning, and generate answers with interpretable decision traces. Applied to Qwen2.5-7B-Instruct and Llama3.1-8B-Instruct, extensive experiments across multiple baselines show 10-30% accuracy improvements on three multi-hop QA datasets, comparable to advanced closed-source LLMs (e.g., OpenAI o1, DeepSeek R1). Further analyses show that ARENA generalizes well to unseen datasets and tasks. Our models and codes are publicly released.

Transparent and Robust RAG: Adaptive-Reward Reinforcement Learning for Decision Traceability

TL;DR

ARENA introduces a transparent and robust RL-based RAG framework that couples a structured evidence–reasoning–answer protocol with KL-divergence stabilization using the estimator and adaptive, task-specific rewards. The approach enforces auditable grounding by requiring explicit <relevance> indices and reasoning traces, while stabilizing policy optimization via GRPO with the stabilized KL term. Four reward components—format, accuracy, relevance, and bonus—drive well-formed, evidence-grounded outputs, achieving 10–30% gains on three multi-hop QA benchmarks and generalizing across models and retrieval settings. The work demonstrates that smaller open-source backbones can approach or match some closed-source models in multi-hop reasoning when trained with ARENA, and provides extensive ablation and cross-task analyses along with public code release.

Abstract

Retrieval-Augmented Generation (RAG) delivers substantial value in knowledge-intensive applications. Many recent works use reinforcement learning (RL) to elicit strong reasoning in RAG generators. However, two key challenges remain unresolved: (1) Transparency: most prior methods do not explicitly indicate which references are actually used during the reasoning that leads to the final answer, limiting interpretability and visibility; (2) Stability: the KL divergence estimator used in existing RL-based approaches may cause gradient spikes, leading to unstable training. To address these challenges, we propose Adaptive-Rewarded Evidence Navigation Agent (ARENA), a transparent and robust RAG generator framework trained via RL with designed rewards. Based on our structured protocol, KL divergence stabilization, and adaptive reward calculation modules, ARENA enables the RAG generator to identify key evidence, perform structured reasoning, and generate answers with interpretable decision traces. Applied to Qwen2.5-7B-Instruct and Llama3.1-8B-Instruct, extensive experiments across multiple baselines show 10-30% accuracy improvements on three multi-hop QA datasets, comparable to advanced closed-source LLMs (e.g., OpenAI o1, DeepSeek R1). Further analyses show that ARENA generalizes well to unseen datasets and tasks. Our models and codes are publicly released.
Paper Structure (47 sections, 11 equations, 5 figures, 13 tables)

This paper contains 47 sections, 11 equations, 5 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. RAG Foundations and the Transparency Gap
  4. Search-Centric RAG: Progress and Remaining Stability Issues
  5. ARENA: Adaptive-Rewarded Evidence Navigation Agent
  6. Framework formalization
  7. Reinforcement learning with stability enhancements
  8. Adaptive reward design
  9. Format Reward $R_{\mathrm{format}}(o_i \mid q)$.
  10. Accuracy Reward $R_{\mathrm{accuracy}}(o_i \mid q)$.
  11. Relevance Reward $R_{\mathrm{relevance}}(o_i \mid q)$.
  12. Bonus Reward $R_{\mathrm{bonus}}(o_i \mid q)$.
  13. Experimental Setups
  14. Datasets
  15. Evaluation Metrics
  16. ...and 32 more sections

Figures (5)

  • Figure 1: Overview of the Adaptive-Rewarded Evidence Navigation Agent (ARENA) framework. The system is composed of three key components: ① Structured Protocol, where the policy model generates a multi-part response including selected evidence, reasoning analysis, and a final answer; ② KL Stabilization, we modify the KL divergence estimator to improve stability; ③ Adaptive Reward Calculation, where the model outputs are evaluated on four axes—format, accuracy, relevance, and bonus—to provide interpretable and fine-grained training signals.
  • Figure 2: Evaluation results on 2Wiki under real-world local search and web search scenarios. ARENA-7B consistently achieves the best performance across all metrics and search modalities, demonstrating its superior generalization and reasoning capabilities in multi-hop question answering.
  • Figure 3: Radar chart comparing six variants across five evaluation metrics on HotPotQA. ARENA shows consistent improvements in output quality and interpretability.
  • Figure 4: A Case Study of Naive GRPO and ARENA on the HotpotQA. ARENA correctly extracts relevant paragraphs, provides a structured explanation, and delivers the right answer.
  • Figure 5: Training KL divergence curves for stabilized vs. unstabilized variants. ARENA's stabilized KL (left) leads to smoother convergence, while the standard estimator (right) suffers from extreme spikes.