LatentMem: Customizing Latent Memory for Multi-Agent Systems

TL;DR

LatentMem introduces a learnable, role-aware memory framework for multi-agent systems that uses a lightweight experience bank and a learnable memory composer to distill raw interactions into compact latent memories. Trained end-to-end via Latent Memory Policy Optimization (LMPO), the system propagates task rewards through memories to improve agent reasoning without altering underlying architectures. Across six benchmarks and four MAS frameworks, LatentMem achieves substantial performance gains, improves generalization to unseen domains, and reduces token and inference costs, addressing memory homogenization and information overload. The approach promises robust, scalable memory customization for real-world MAS deployments, while highlighting ethical considerations and the importance of safeguards in powerful memory-enabled systems.

Abstract

Large language model (LLM)-powered multi-agent systems (MAS) demonstrate remarkable collective intelligence, wherein multi-agent memory serves as a pivotal mechanism for continual adaptation. However, existing multi-agent memory designs remain constrained by two fundamental bottlenecks: (i) memory homogenization arising from the absence of role-aware customization, and (ii) information overload induced by excessively fine-grained memory entries. To address these limitations, we propose LatentMem, a learnable multi-agent memory framework designed to customize agent-specific memories in a token-efficient manner. Specifically, LatentMem comprises an experience bank that stores raw interaction trajectories in a lightweight form, and a memory composer that synthesizes compact latent memories conditioned on retrieved experience and agent-specific contexts. Further, we introduce Latent Memory Policy Optimization (LMPO), which propagates task-level optimization signals through latent memories to the composer, encouraging it to produce compact and high-utility representations. Extensive experiments across diverse benchmarks and mainstream MAS frameworks show that LatentMem achieves a performance gain of up to $19.36$% over vanilla settings and consistently outperforms existing memory architectures, without requiring any modifications to the underlying frameworks.

Figures (10)

• Figure 1: The paradigm comparison between existing multi-agent memory and LatentMem. Instead of relying on handcrafted memory units, LatentMem extracts agent-specific memories from the latent space by combining raw trajectories with agent profiles.
• Figure 2: Overview of LatentMem. The framework proceeds as follows: (1) retrieve relevant trajectories from the experience bank; (2) compress them with agent role profiles into latent memories via the LMPO-trained memory composer; (3) inject these memories into agent reasoning processes without altering the agent architectures; and (4) store new trajectories for continual improvement.
• Figure 3: Time and token consumption of LatentMem. Each panel shows the trade-off between performance and resource cost under different memory architectures: the top row plots performance versus time, the bottom row plots performance versus token cost. Circle area reflects relative resource consumption.
• Figure 4: t-SNE visualization of latent memories generated by LatentMem across different datasets and MAS frameworks.
• Figure 5: Evolution of cumulative accuracy (reward) across question indices. The cumulative accuracy at index $i$ is defined as the average accuracy (reward) over the first $i$ questions.