Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mem-T: Densifying Rewards for Long-Horizon Memory Agents

Yanwei Yue, Guibin Zhang, Boci Peng, Xuanbo Fan, Jiaxin Guo, Qiankun Li, Yan Zhang

TL;DR

Mem-T addresses the challenge of optimizing long-horizon memory management under sparse rewards by introducing a hierarchical memory database and a dual-track training framework. The core innovation, Memory Operation Tree-guided GRPO (MoT-GRPO), densifies terminal rewards through node-wise backpropagation and hindsight credit assignment, enabling joint optimization of memory construction and retrieval. Empirical results show Mem-T achieving state-of-the-art performance across in-domain and out-of-domain benchmarks with favorable accuracy-efficiency trade-offs, including substantial reductions in per-query token usage. The approach demonstrates strong generalization and ablation-supported robustness, offering a pathway toward self-evolving, lifelong memory-enabled agents.

Abstract

Memory agents, which depart from predefined memory-processing pipelines by endogenously managing the processing, storage, and retrieval of memories, have garnered increasing attention for their autonomy and adaptability. However, existing training paradigms remain constrained: agents often traverse long-horizon sequences of memory operations before receiving sparse and delayed rewards, which hinders truly end-to-end optimization of memory management policies. To address this limitation, we introduce Mem-T, an autonomous memory agent that interfaces with a lightweight hierarchical memory database to perform dynamic updates and multi-turn retrieval over streaming inputs. To effectively train long-horizon memory management capabilities, we further propose MoT-GRPO, a tree-guided reinforcement learning framework that transforms sparse terminal feedback into dense, step-wise supervision via memory operation tree backpropagation and hindsight credit assignment, thereby enabling the joint optimization of memory construction and retrieval. Extensive experiments demonstrate that Mem-T is (1) high-performing, surpassing frameworks such as A-Mem and Mem0 by up to $14.92\%$, and (2) economical, operating on a favorable accuracy-efficiency Pareto frontier and reducing inference tokens per query by $\sim24.45\%$ relative to GAM without sacrificing performance.

Mem-T: Densifying Rewards for Long-Horizon Memory Agents

TL;DR

Mem-T addresses the challenge of optimizing long-horizon memory management under sparse rewards by introducing a hierarchical memory database and a dual-track training framework. The core innovation, Memory Operation Tree-guided GRPO (MoT-GRPO), densifies terminal rewards through node-wise backpropagation and hindsight credit assignment, enabling joint optimization of memory construction and retrieval. Empirical results show Mem-T achieving state-of-the-art performance across in-domain and out-of-domain benchmarks with favorable accuracy-efficiency trade-offs, including substantial reductions in per-query token usage. The approach demonstrates strong generalization and ablation-supported robustness, offering a pathway toward self-evolving, lifelong memory-enabled agents.

Abstract

Memory agents, which depart from predefined memory-processing pipelines by endogenously managing the processing, storage, and retrieval of memories, have garnered increasing attention for their autonomy and adaptability. However, existing training paradigms remain constrained: agents often traverse long-horizon sequences of memory operations before receiving sparse and delayed rewards, which hinders truly end-to-end optimization of memory management policies. To address this limitation, we introduce Mem-T, an autonomous memory agent that interfaces with a lightweight hierarchical memory database to perform dynamic updates and multi-turn retrieval over streaming inputs. To effectively train long-horizon memory management capabilities, we further propose MoT-GRPO, a tree-guided reinforcement learning framework that transforms sparse terminal feedback into dense, step-wise supervision via memory operation tree backpropagation and hindsight credit assignment, thereby enabling the joint optimization of memory construction and retrieval. Extensive experiments demonstrate that Mem-T is (1) high-performing, surpassing frameworks such as A-Mem and Mem0 by up to , and (2) economical, operating on a favorable accuracy-efficiency Pareto frontier and reducing inference tokens per query by relative to GAM without sacrificing performance.
Paper Structure (47 sections, 19 equations, 9 figures, 5 tables)

This paper contains 47 sections, 19 equations, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Memory Agent Architectures.
  4. Reinforcement Learning for Memory Agents.
  5. Method
  6. Mem-T Workflow
  7. Hierarchical Memory Definition.
  8. Memory Operation Pipeline.
  9. MoT-GRPO for Memory Retrieval
  10. Memory Operation Tree Construction.
  11. Node-wise Reward Backpropagation.
  12. Dual-Scale Advantage Estimation.
  13. Optimization Objective.
  14. MoT-GRPO for Memory Construction
  15. Hindsight Credit Assignment.
  16. ...and 32 more sections

Figures (9)

  • Figure 1: The paradigm comparison between the previous trainable memory agent and Mem-T.
  • Figure 2: The overall framework of our proposed Mem-T.
  • Figure 3: The comparison of the performance and inference cost on the LoCoMo dataset. Different shapes of the scatter points represent various types of baselines.
  • Figure 4: The comparison of the performance and inference cost on the HotpotQA dataset. Different shapes of the scatter points represent various types of baselines.
  • Figure 5: (Left) Parameter sensitivity analysis on the max inference retrieval steps on the LoCoMo; (Right) Parameter sensitivity analysis on the number of operation trees per query($G$) when training with MoT-GRPO on the LoCoMo and HotpotQA dataset.
  • ...and 4 more figures