Fine-Mem: Fine-Grained Feedback Alignment for Long-Horizon Memory Management

TL;DR

Fine-Mem tackles reward sparsity and credit assignment in long-horizon memory management for LLM agents by introducing two components: Chunk-level Step Reward (CSR), which provides immediate, chunk-specific supervision via QA pairs, and Evidence-Anchored Reward Attribution (EARA), which redistributes the global reward to memory operations based on evidential contribution. The method is trained with Group Relative Policy Optimization (GRPO) under a hybrid reward that also includes formatting and compression incentives. Empirical results on Memalpha and MemoryAgentBench show that Fine-Mem consistently outperforms seven baselines, achieving average gains of 4.4% and 7.2% respectively, while maintaining compact memory footprints. The work demonstrates strong generalization across reasoning backbones and memory architectures, underscoring the practical impact of dense, operation-aware feedback for long-horizon memory systems.

Abstract

Effective memory management is essential for large language model agents to navigate long-horizon tasks. Recent research has explored using Reinforcement Learning to develop specialized memory manager agents. However, existing approaches rely on final task performance as the primary reward, which results in severe reward sparsity and ineffective credit assignment, providing insufficient guidance for individual memory operations. To this end, we propose Fine-Mem, a unified framework designed for fine-grained feedback alignment. First, we introduce a Chunk-level Step Reward to provide immediate step-level supervision via auxiliary chunk-specific question answering tasks. Second, we devise Evidence-Anchored Reward Attribution to redistribute global rewards by anchoring credit to key memory operations, based on the specific memory items utilized as evidence in reasoning. Together, these components enable stable policy optimization and align local memory operations with the long-term utility of memory. Experiments on Memalpha and MemoryAgentBench demonstrate that Fine-Mem consistently outperforms strong baselines, achieving superior success rates across various sub-tasks. Further analysis reveals its adaptability and strong generalization capabilities across diverse model configurations and backbones.

Figures (10)

• Figure 1: Comparison of memory management paradigms. Left: Workflow-based mode relies on pre-defined pipelines and strong LLMs, suffering from high overhead and poor scalability. Right: Training-based mode utilizes a specialized manager optimized via RL, improving effectiveness but hindered by sparse rewards and ineffective credit assignment problems.
• Figure 2: An overview of Fine-Mem. Left: The overall training framework. Right: Two core components designed to enhance training: (1) Chunk-level Step Reward (§\ref{['meth:chunk_step_reward']}), which addresses reward sparsity by generating chunk-level QA tasks to provide step-level feedback for memory operations; (2) Evidence-Anchored Reward Attribution (§\ref{['meth:EARA']}), which resolves the credit assignment challenge by redistributing global rewards back to specific rollout steps.
• Figure 3: Ablation study on the hyperparameter $\beta$ in Evidence-Anchored Reward Attribution on Memalpha and MemoryAgentBench (MAB.)
• Figure 4: Performance comparison of different Memory Managers combined with varying Reasoning Models on the Memalpha dataset.
• Figure 5: Performance comparison of different Memory Managers combined with varying Reasoning Models on MemoryAgentBench dataset.