Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Memory as Action: Autonomous Context Curation for Long-Horizon Agentic Tasks

Yuxiang Zhang, Jiangming Shu, Ye Ma, Xueyuan Lin, Shangxi Wu, Jitao Sang

TL;DR

A new algorithm is proposed, Dynamic Context Policy Optimization, which enables stable end-to-end reinforcement learning by segmenting trajectories at memory action points and applying trajectory-level advantages to the resulting action segments, demonstrating that jointly optimizing for task reasoning and memory management in an end-to-end fashion not only reduces overall computational consumption but also improves task performance.

Abstract

Large Language Models face challenges in long-horizon agentic tasks as their constrained memory is easily overwhelmed by distracting or irrelevant context. Existing working memory methods typically rely on external, heuristic mechanisms that are decoupled from the agent's core policy. In this work, we reframe working memory management as a learnable, intrinsic capability. We propose a novel framework, Memory-as-Action, where an agent actively manages its working memory by executing explicit editing operations as part of a unified policy. This formulation allows an agent, trained via reinforcement learning, to balance memory curation against long-term task objectives under given resource constraints. However, such memory editing actions break the standard assumption of a continuously growing prefix in LLM interactions, leading to what we call trajectory fractures. These non-prefix changes disrupt the causal continuity required by standard policy gradient methods, making those methods inapplicable. To address this, we propose a new algorithm, Dynamic Context Policy Optimization, which enables stable end-to-end reinforcement learning by segmenting trajectories at memory action points and applying trajectory-level advantages to the resulting action segments. Our results demonstrate that jointly optimizing for task reasoning and memory management in an end-to-end fashion not only reduces overall computational consumption but also improves task performance, driven by adaptive context curation strategies tailored to the model's intrinsic capabilities.

Memory as Action: Autonomous Context Curation for Long-Horizon Agentic Tasks

TL;DR

A new algorithm is proposed, Dynamic Context Policy Optimization, which enables stable end-to-end reinforcement learning by segmenting trajectories at memory action points and applying trajectory-level advantages to the resulting action segments, demonstrating that jointly optimizing for task reasoning and memory management in an end-to-end fashion not only reduces overall computational consumption but also improves task performance.

Abstract

Large Language Models face challenges in long-horizon agentic tasks as their constrained memory is easily overwhelmed by distracting or irrelevant context. Existing working memory methods typically rely on external, heuristic mechanisms that are decoupled from the agent's core policy. In this work, we reframe working memory management as a learnable, intrinsic capability. We propose a novel framework, Memory-as-Action, where an agent actively manages its working memory by executing explicit editing operations as part of a unified policy. This formulation allows an agent, trained via reinforcement learning, to balance memory curation against long-term task objectives under given resource constraints. However, such memory editing actions break the standard assumption of a continuously growing prefix in LLM interactions, leading to what we call trajectory fractures. These non-prefix changes disrupt the causal continuity required by standard policy gradient methods, making those methods inapplicable. To address this, we propose a new algorithm, Dynamic Context Policy Optimization, which enables stable end-to-end reinforcement learning by segmenting trajectories at memory action points and applying trajectory-level advantages to the resulting action segments. Our results demonstrate that jointly optimizing for task reasoning and memory management in an end-to-end fashion not only reduces overall computational consumption but also improves task performance, driven by adaptive context curation strategies tailored to the model's intrinsic capabilities.
Paper Structure (22 sections, 3 equations, 4 figures, 1 table, 2 algorithms)

This paper contains 22 sections, 3 equations, 4 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Method
  3. Overview
  4. Formalizing Context Management as a Decision Process
  5. Dynamic Context Policy Optimization
  6. Outcome Reward
  7. Trajectory Segmentation and Sampling
  8. Trajectory-Level Policy Optimization
  9. Experiments & Results
  10. Datasets
  11. Baselines
  12. Implementation Details
  13. Memory Management Implementation
  14. Cold-start via Segmented Supervised Finetuning
  15. DCPO Hyperparameters
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Comparison between MemAct and conventional memory management. The left side illustrates a representative design in existing systems, where memory operations—such as selection, compression, and summarization—are governed by handcrafted heuristics or external controllers. These behaviors remain decoupled from the agent's core decision-making process. In contrast, the Memory-as-Action (MemAct) framework integrates such operations into the policy itself, enabling the agent to learn when and how to edit its own working memory as part of a unified decision loop. This formulation supports goal-directed, policy-driven memory management.
  • Figure 2: Model performance on the Multi-Objective QA dataset.(a) Per-objective accuracy versus average input tokens per round. The top-left quadrant represents higher accuracy with greater context efficiency. (b) Total input tokens per query versus the average number of function calls, illustrating the end-to-end efficiency of different strategies.
  • Figure 3: Comparative performance on the Multi-Objective QA dataset. The left panel shows task accuracy and the right panel shows the average number of task-related tool calls, broken down by the number of objectives per query. Models are sorted by their overall average accuracy. Memory actions are excluded from the tool call count.
  • Figure 4: Tool usage by objective count.(a) Average external-tool calls. (b) Average memory-management calls. Rows: models; columns: objectives. Independent color scales per panel.