Collaborative Multi-Agent Optimization for Personalized Memory System

Abstract

Memory systems are crucial to personalized LLMs by mitigating the context window limitation in capturing long-term user-LLM conversations. Typically, such systems leverage multiple agents to handle multi-granular memory construction and personalized memory retrieval tasks. To optimize the system, existing methods focus on specializing agents on their local tasks independently via prompt engineering or fine-tuning. However, they overlook cross-agent collaboration, where independent optimization on local agents hardly guarantees the global system performance. To address this issue, we propose a Collaborative Reinforcement Learning Framework for Multi-Agent Memory Systems (CoMAM), jointly optimizing local agents to facilitate collaboration. Specifically, we regularize agents' execution as a sequential Markov decision process (MDP) to embed inter-agent dependencies into the state transition, yielding both local task rewards (e.g., information coverage for memory construction) and global rewards (i.e., query-answer accuracy). Then, we quantify each agent's contribution via group-level ranking consistency between local and global rewards, treating them as adaptive weights to assign global credit and integrate local-global rewards. Each agent is optimized by these integrated rewards, aligning local improvements with the global performance. Experiments show CoMAM outperforms leading memory systems, validating the efficacy of our proposed collaborative reinforcement learning for joint optimization.

Figures (6)

• Figure 1: Construction and retrieval agents, optimized on local tasks independently, yield lower global system performance than those under joint optimization.
• Figure 2: Illustration of challenges for joint optimization.
• Figure 3: The overview of our proposed framework, CoMAM, which regularizes agents' execution as MDP trajectories for joint RL optimization and fosters collaboration via adaptive credit assignment to achieve local-global alignment.
• Figure 4: Detailed performance of different methods across seven question types on the PersonaMem benchmark across three history length settings (i.e., 32K, 128K, and 1M). The number of question types is consistent with those listed in Section \ref{['sec: exp_settings']}
• Figure 5: The left demonstrates CoMAM’s sensitivity to the credit assignment weight, and the right shows the credit assignment’s impact on each local agent’s performance on the 32K setting.