Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GlobalRAG: Enhancing Global Reasoning in Multi-hop Question Answering via Reinforcement Learning

Jinchang Luo, Mingquan Cheng, Fan Wan, Ni Li, Xiaoling Xia, Shuangshuang Tian, Tingcheng Bian, Haiwei Wang, Haohuan Fu, Yan Tao

TL;DR

GlobalRAG tackles two core issues in multi-hop QA: lack of global planning and unfaithful execution. It introduces planning-aware reinforcement learning, leveraging subgoal decomposition, golden trajectories, and a suite of rewards (planning quality, subgoal completion, outcome, and format) together with progressive weight annealing to stabilize training. A teacher-generated golden trajectory guides global planning, while GRPO-style rollout optimizes coherent reasoning and faithful evidence use. Across five benchmarks with a small data budget, GlobalRAG delivers consistent EM and F1 gains and generalizes across backbones, demonstrating the practical impact of plan-centric RL for complex reasoning.

Abstract

Reinforcement learning has recently shown promise in improving retrieval-augmented generation (RAG). Despite these advances, its effectiveness in multi-hop question answering (QA) remains limited by two fundamental limitations: (i) global planning absence to structure multi-step reasoning, and (ii) unfaithful execution, which hinders effective query formulation and consistent use of retrieved evidence. We propose GlobalRAG, a reinforcement learning framework designed to enhance global reasoning in multi-hop QA. GlobalRAG decomposes questions into subgoals, coordinates retrieval with reasoning, and refines evidence iteratively. To guide this process, we introduce Planning Quality Reward and SubGoal Completion Reward, which encourage coherent planning and reliable subgoal execution. In addition, a progressive weight annealing strategy balances process-oriented and outcome-based objectives. Extensive experiments on both in-domain and out-of-domain benchmarks demonstrate that GlobalRAG significantly outperforms strong baselines while using only 8k training data (42% of the training data used by strong baselines), achieving average improvements of 14.2% in both EM and F1.

GlobalRAG: Enhancing Global Reasoning in Multi-hop Question Answering via Reinforcement Learning

TL;DR

GlobalRAG tackles two core issues in multi-hop QA: lack of global planning and unfaithful execution. It introduces planning-aware reinforcement learning, leveraging subgoal decomposition, golden trajectories, and a suite of rewards (planning quality, subgoal completion, outcome, and format) together with progressive weight annealing to stabilize training. A teacher-generated golden trajectory guides global planning, while GRPO-style rollout optimizes coherent reasoning and faithful evidence use. Across five benchmarks with a small data budget, GlobalRAG delivers consistent EM and F1 gains and generalizes across backbones, demonstrating the practical impact of plan-centric RL for complex reasoning.

Abstract

Reinforcement learning has recently shown promise in improving retrieval-augmented generation (RAG). Despite these advances, its effectiveness in multi-hop question answering (QA) remains limited by two fundamental limitations: (i) global planning absence to structure multi-step reasoning, and (ii) unfaithful execution, which hinders effective query formulation and consistent use of retrieved evidence. We propose GlobalRAG, a reinforcement learning framework designed to enhance global reasoning in multi-hop QA. GlobalRAG decomposes questions into subgoals, coordinates retrieval with reasoning, and refines evidence iteratively. To guide this process, we introduce Planning Quality Reward and SubGoal Completion Reward, which encourage coherent planning and reliable subgoal execution. In addition, a progressive weight annealing strategy balances process-oriented and outcome-based objectives. Extensive experiments on both in-domain and out-of-domain benchmarks demonstrate that GlobalRAG significantly outperforms strong baselines while using only 8k training data (42% of the training data used by strong baselines), achieving average improvements of 14.2% in both EM and F1.
Paper Structure (27 sections, 13 equations, 4 figures, 8 tables)

This paper contains 27 sections, 13 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. Method
  5. Rollout Retrieval Enhanced GRPO for Multi-hop QA
  6. Golden Trajectory Generation with Global Planning
  7. Global Planning–Aware Reward Design
  8. Planning Quality Reward
  9. Subgoal Completion Reward
  10. Outcome Reward
  11. Format Reward
  12. Progressive weight annealing strategy
  13. Experiments
  14. Experimental Setup
  15. Main Results
  16. ...and 12 more sections

Figures (4)

  • Figure 1: (top) Example of global planning absence and unfaithful execution. (bottom left) The most frequent failure of Search-R1 is missing the correct document. (bottom right) Error distribution where these two issues account for over 94% of failures.
  • Figure 2: Overview of GlobalRAG. The top illustrates rollout retrieval–enhanced GRPO with a search engine; the bottom shows golden trajectories constructed by a teacher model; and the middle depicts the global planning–aware reward design used to update the policy model.
  • Figure 3: Comparison of average retrieval counts across datasets for Search-R1, StepSearch and GlobalRAG.
  • Figure 4: The prompt used for trajectory generation