Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforced Internal-External Knowledge Synergistic Reasoning for Efficient Adaptive Search Agent

Ziyang Huang, Xiaowei Yuan, Yiming Ju, Jun Zhao, Kang Liu

TL;DR

The paper tackles hallucinations and latency in retrieval-augmented reasoning by introducing IKEA, an RL-trained agent that explicitly learns its own knowledge boundary and prioritizes internal parametric knowledge, using external search only when needed. It combines a knowledge-boundary aware reward with a specially constructed training dataset to foster adaptive, synergistic use of internal and external knowledge. Empirical results across single- and multi-hop tasks show IKEA reduces retrieval frequency while achieving higher exact-match performance than baselines, with strong generalization to out-of-distribution data. The work advances efficient, knowledge-boundary aware decision-making for adaptive search in LLM-based agents, with implications for faster, more reliable knowledge-intensive reasoning.

Abstract

Retrieval-augmented generation (RAG) is a common strategy to reduce hallucinations in Large Language Models (LLMs). While reinforcement learning (RL) can enable LLMs to act as search agents by activating retrieval capabilities, existing ones often underutilize their internal knowledge. This can lead to redundant retrievals, potential harmful knowledge conflicts, and increased inference latency. To address these limitations, an efficient and adaptive search agent capable of discerning optimal retrieval timing and synergistically integrating parametric (internal) and retrieved (external) knowledge is in urgent need. This paper introduces the Reinforced Internal-External Knowledge Synergistic Reasoning Agent (IKEA), which could indentify its own knowledge boundary and prioritize the utilization of internal knowledge, resorting to external search only when internal knowledge is deemed insufficient. This is achieved using a novel knowledge-boundary aware reward function and a knowledge-boundary aware training dataset. These are designed for internal-external knowledge synergy oriented RL, incentivizing the model to deliver accurate answers, minimize unnecessary retrievals, and encourage appropriate external searches when its own knowledge is lacking. Evaluations across multiple knowledge reasoning tasks demonstrate that IKEA significantly outperforms baseline methods, reduces retrieval frequency significantly, and exhibits robust generalization capabilities.

Reinforced Internal-External Knowledge Synergistic Reasoning for Efficient Adaptive Search Agent

TL;DR

The paper tackles hallucinations and latency in retrieval-augmented reasoning by introducing IKEA, an RL-trained agent that explicitly learns its own knowledge boundary and prioritizes internal parametric knowledge, using external search only when needed. It combines a knowledge-boundary aware reward with a specially constructed training dataset to foster adaptive, synergistic use of internal and external knowledge. Empirical results across single- and multi-hop tasks show IKEA reduces retrieval frequency while achieving higher exact-match performance than baselines, with strong generalization to out-of-distribution data. The work advances efficient, knowledge-boundary aware decision-making for adaptive search in LLM-based agents, with implications for faster, more reliable knowledge-intensive reasoning.

Abstract

Retrieval-augmented generation (RAG) is a common strategy to reduce hallucinations in Large Language Models (LLMs). While reinforcement learning (RL) can enable LLMs to act as search agents by activating retrieval capabilities, existing ones often underutilize their internal knowledge. This can lead to redundant retrievals, potential harmful knowledge conflicts, and increased inference latency. To address these limitations, an efficient and adaptive search agent capable of discerning optimal retrieval timing and synergistically integrating parametric (internal) and retrieved (external) knowledge is in urgent need. This paper introduces the Reinforced Internal-External Knowledge Synergistic Reasoning Agent (IKEA), which could indentify its own knowledge boundary and prioritize the utilization of internal knowledge, resorting to external search only when internal knowledge is deemed insufficient. This is achieved using a novel knowledge-boundary aware reward function and a knowledge-boundary aware training dataset. These are designed for internal-external knowledge synergy oriented RL, incentivizing the model to deliver accurate answers, minimize unnecessary retrievals, and encourage appropriate external searches when its own knowledge is lacking. Evaluations across multiple knowledge reasoning tasks demonstrate that IKEA significantly outperforms baseline methods, reduces retrieval frequency significantly, and exhibits robust generalization capabilities.
Paper Structure (28 sections, 3 equations, 4 figures, 4 tables)

This paper contains 28 sections, 3 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Multi-turn Reinforcement Learning with Verifiable Reward for Large Language Model
  4. Knowledge Boundary of Large Language Model
  5. Method
  6. Basic Setting
  7. IEKA: Reinforced Internal-External Knowledge Synergistic Reasoning Agent
  8. IKEA Agent Prompt Template
  9. Knowledge-boundary Aware Reward Design
  10. Knowledge-boundary Aware Dataset Construction
  11. Experiment
  12. Setting
  13. Overall Results
  14. Baselines without parameter updates struggle to effectively synergize internal and external knowledge.
  15. Reinforcement learning baselines can effectively activate the model's capacity to solely utilize its internal knowledge or solely utilize external knowledge accessed via retrieval.
  16. ...and 13 more sections

Figures (4)

  • Figure 1: The top of the figure illustrates the training process for Multi-turn Reinforcement Learning with Verifiable Reward for LLM-Agent. In the middle is Search-R1, and at the very bottom is IKEA. Search-R1 and IKEA are special types of LLM-agents. We highlight the differences from the training of general LLM-agents, and to save space, we have omitted the common parts, such as the calculation of KL and Advantage.
  • Figure 2: The training log of IKEA-3B-Zero, IKEA-3B, IKEA-7B-Zero and IKEA-7B. We show the curve of number of valid searches, response length and trainign rewards.
  • Figure 3: The training logs of different reward design. We show the curve of number of valid searches, response length and trainign rewards.
  • Figure 4: The training logs of different the difficulty of training datasets. We show the curve of number of valid searches, response length and trainign rewards.