Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dep-Search: Learning Dependency-Aware Reasoning Traces with Persistent Memory

Yanming Liu, Xinyue Peng, Zixuan Yan, Yanxin Shen, Wenjie Xu, Yuefeng Huang, Xinyi Wang, Jiannan Cao, Jianwei Yin, Xuhong Zhang

TL;DR

Dep-Search targets dependency-aware, memory-enabled reasoning for multi-hop QA by coupling explicit sub-question decomposition with a persistent memory system and trajectory-level RL via GRPO. It introduces explicit control tokens to manage Decompose, Retrieve, Memory, and Conclusion steps, a fixed-capacity LRU memory, and a reward design that balances answer quality with efficient search. Across six QA datasets and two model scales, Dep-Search shows consistent improvements, with larger gains on complex multi-hop tasks and ablations highlighting the central role of memory and dependency Modeling. The work demonstrates a practical path toward scalable, reusable reasoning traces in LLMs and motivates further integration with diverse knowledge sources and dynamic memory strategies.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks, particularly when augmented with search mechanisms that enable systematic exploration of external knowledge bases. The field has evolved from traditional retrieval-augmented generation (RAG) frameworks to more sophisticated search-based frameworks that orchestrate multi-step reasoning through explicit search strategies. However, existing search frameworks still rely heavily on implicit natural language reasoning to determine search strategies and how to leverage retrieved information across reasoning steps. This reliance on implicit reasoning creates fundamental challenges for managing dependencies between sub-questions, efficiently reusing previously retrieved knowledge, and learning optimal search strategies through reinforcement learning. To address these limitations, we propose Dep-Search, a dependency-aware search framework that advances beyond existing search frameworks by integrating structured reasoning, retrieval, and persistent memory through GRPO. Dep-Search introduces explicit control mechanisms that enable the model to decompose questions with dependency relationships, retrieve information when needed, access previously stored knowledge from memory, and summarize long reasoning contexts into reusable memory entries. Through extensive experiments on seven diverse question answering datasets, we demonstrate that Dep-Search significantly enhances LLMs' ability to tackle complex multi-hop reasoning tasks, achieving substantial improvements over strong baselines across different model scales.

Dep-Search: Learning Dependency-Aware Reasoning Traces with Persistent Memory

TL;DR

Dep-Search targets dependency-aware, memory-enabled reasoning for multi-hop QA by coupling explicit sub-question decomposition with a persistent memory system and trajectory-level RL via GRPO. It introduces explicit control tokens to manage Decompose, Retrieve, Memory, and Conclusion steps, a fixed-capacity LRU memory, and a reward design that balances answer quality with efficient search. Across six QA datasets and two model scales, Dep-Search shows consistent improvements, with larger gains on complex multi-hop tasks and ablations highlighting the central role of memory and dependency Modeling. The work demonstrates a practical path toward scalable, reusable reasoning traces in LLMs and motivates further integration with diverse knowledge sources and dynamic memory strategies.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks, particularly when augmented with search mechanisms that enable systematic exploration of external knowledge bases. The field has evolved from traditional retrieval-augmented generation (RAG) frameworks to more sophisticated search-based frameworks that orchestrate multi-step reasoning through explicit search strategies. However, existing search frameworks still rely heavily on implicit natural language reasoning to determine search strategies and how to leverage retrieved information across reasoning steps. This reliance on implicit reasoning creates fundamental challenges for managing dependencies between sub-questions, efficiently reusing previously retrieved knowledge, and learning optimal search strategies through reinforcement learning. To address these limitations, we propose Dep-Search, a dependency-aware search framework that advances beyond existing search frameworks by integrating structured reasoning, retrieval, and persistent memory through GRPO. Dep-Search introduces explicit control mechanisms that enable the model to decompose questions with dependency relationships, retrieve information when needed, access previously stored knowledge from memory, and summarize long reasoning contexts into reusable memory entries. Through extensive experiments on seven diverse question answering datasets, we demonstrate that Dep-Search significantly enhances LLMs' ability to tackle complex multi-hop reasoning tasks, achieving substantial improvements over strong baselines across different model scales.
Paper Structure (31 sections, 18 equations, 4 figures, 3 tables, 1 algorithm)

This paper contains 31 sections, 18 equations, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Agentic Reinforcement Learning
  4. Agentic Memory
  5. Methodology
  6. Problem Overview
  7. Data Collection and Design
  8. Memory Module
  9. RL with Dep-Search
  10. Reward Model
  11. Experimental Setup
  12. Datasets
  13. Baselines
  14. Models and Metrics
  15. Experiments
  16. ...and 16 more sections

Figures (4)

  • Figure 1: Overview of the Dep-Search framework. The agent decomposes questions into dependent sub-questions, retrieves relevant information, accesses stored knowledge from memory, and synthesizes answers through trajectory-level reinforcement learning.
  • Figure 2: Reward function threshold sensitivity analysis on 2WikiMHQA.
  • Figure 3: Action call frequency per question across different datasets on Qwen2.5-7B-Instruct.
  • Figure 4: Memory capacity sensitivity analysis on 2WikiMHQA.