MemAdapter: Fast Alignment across Agent Memory Paradigms via Generative Subgraph Retrieval

TL;DR

MemAdapter addresses the fragmentation of memory paradigms in LLM-based agents by proposing a paradigm-agnostic, plug-and-play memory retrieval framework. It learns a generative subgraph retriever within a unified memory space and then rapidly aligns to unseen memory paradigms through lightweight contrastive learning, enabling cross-paradigm retrieval and fusion with minimal compute. Across three memory-intensive QA benchmarks and multiple model scales, MemAdapter consistently outperforms strong baseline memory systems and achieves cross-paradigm alignment in under 13 minutes on a single GPU. The work demonstrates practical benefits for memory-augmented agents, including efficient fusion of heterogeneous memories and scalable deployment without extensive architectural redesign.

Abstract

Memory mechanism is a core component of LLM-based agents, enabling reasoning and knowledge discovery over long-horizon contexts. Existing agent memory systems are typically designed within isolated paradigms (e.g., explicit, parametric, or latent memory) with tightly coupled retrieval methods that hinder cross-paradigm generalization and fusion. In this work, we take a first step toward unifying heterogeneous memory paradigms within a single memory system. We propose MemAdapter, a memory retrieval framework that enables fast alignment across agent memory paradigms. MemAdapter adopts a two-stage training strategy: (1) training a generative subgraph retriever from the unified memory space, and (2) adapting the retriever to unseen memory paradigms by training a lightweight alignment module through contrastive learning. This design improves the flexibility for memory retrieval and substantially reduces alignment cost across paradigms. Comprehensive experiments on three public evaluation benchmarks demonstrate that the generative subgraph retriever consistently outperforms five strong agent memory systems across three memory paradigms and agent model scales. Notably, MemAdapter completes cross-paradigm alignment within 13 minutes on a single GPU, achieving superior performance over original memory retrievers with less than 5% of training compute. Furthermore, MemAdapter enables effective zero-shot fusion across memory paradigms, highlighting its potential as a plug-and-play solution for agent memory systems.

Figures (8)

• Figure 1: Comparison between paradigm-specific memory retrieval and paradigm-agnostic generative retrieval.
• Figure 2: MemAdapter performs generative subgraph retrieval from a unified memory space adapted by paradigm-specific alignment modules. Building on an anchored memory paradigm, the first stage introduces a teacher model to provide imitation-based supervision for the retriever. In stage two, the anchored alignment module is used to supervise a lightweight alignment module for the target paradigm via contrastive learning. At inference, aligned memory paradigms can be used and infused in a plug-and-play manner.
• Figure 3: Increasing context information with heterogeneous memory encoding. Context segments at different positions are fused by Eqn. \ref{['eq:downsample']}.
• Figure 4: A t-SNE visualization of heterogeneous memory representations from the unified memory space, before and after cross-paradigm alignment. Memory states for parametric memory are obtained via the Qwen2.5-3B-Instruct agent model.
• Figure 5: A case study of heterogeneous memory fusion.