Beyond RAG for Agent Memory: Retrieval by Decoupling and Aggregation

TL;DR

This work identifies a mismatch between standard RAG retrieval and agent memory, where memories are a bounded, coherent dialogue stream rather than heterogeneous external text. It introduces xMemory, a hierarchical memory framework that disentangles memories into semantic components (Original messages -> Episode -> Semantic -> Theme) and uses a sparsity–semantics objective to guide structure, enabling top-down retrieval that favors diverse, query-relevant themes before expanding to finer details. The method combines a two-stage retrieval (Stage I: representative selection on a kNN graph; Stage II: uncertainty-driven evidence inclusion) with retroactive restructuring to adapt the memory organization as new information arrives. Across LoCoMo and PerLTQA benchmarks and multiple LLM backbones, xMemory yields consistent improvements in answer quality and token efficiency, while delivering more evidence-dense retrieval than RAG plus pruning baselines, demonstrating the practical value of structure-aware memory management for long-horizon agent reasoning.

Abstract

Agent memory systems often adopt the standard Retrieval-Augmented Generation (RAG) pipeline, yet its underlying assumptions differ in this setting. RAG targets large, heterogeneous corpora where retrieved passages are diverse, whereas agent memory is a bounded, coherent dialogue stream with highly correlated spans that are often duplicates. Under this shift, fixed top-$k$ similarity retrieval tends to return redundant context, and post-hoc pruning can delete temporally linked prerequisites needed for correct reasoning. We argue retrieval should move beyond similarity matching and instead operate over latent components, following decoupling to aggregation: disentangle memories into semantic components, organise them into a hierarchy, and use this structure to drive retrieval. We propose xMemory, which builds a hierarchy of intact units and maintains a searchable yet faithful high-level node organisation via a sparsity--semantics objective that guides memory split and merge. At inference, xMemory retrieves top-down, selecting a compact, diverse set of themes and semantics for multi-fact queries, and expanding to episodes and raw messages only when it reduces the reader's uncertainty. Experiments on LoCoMo and PerLTQA across the three latest LLMs show consistent gains in answer quality and token efficiency.

Figures (5)

• Figure 1: From similarity top-$k$ to structured retrieval for agent memory. Agent memory forms a coherent and highly correlated stream, where many spans are near duplicates; similarity top-$k$ retrieval can therefore collapse and retrieve redundant chunks. xMemory organises memories into a hierarchy of intact units and performs structure-aware retrieval to produce a shorter but more answer-sufficient context.
• Figure 2: Overview of xMemory. xMemory couples memory structuring with top-down retrieval to address the mismatch between agent memory and the RAG pipeline. It organises a coherent stream into a hierarchy that disentangles episodic traces into semantic components while preserving intact units. A sparsity--semantics objective guides split and merge to keep the high-level organisation searchable and faithful. At retrieval, xMemory selects a diverse set of relevant themes and semantics to support aggregation reasoning, then expands to episodes and raw messages only when they decrease the reader’s uncertainty, yielding a shorter context with stronger evidence coverage.
• Figure 3: Ablation on LoCoMo with Qwen3-8B. We report BLEU and F1 on the left axis and Token/query on the right axis (lower is better). Memory-only uses our hierarchical memory structure with a simple similarity retriever, but disables both adaptive retrieval stages. +RepSel adds Stage I, which selects representative theme and semantic nodes on the high-level kNN graph. +UncSion adds Stage II, which admits episodic and original evidence only when it reduces the reader’s uncertainty. Ours (Full) combines both stages.
• Figure 4: Evidence hit distribution by question category. Each subplot shows the proportions of retrieved blocks that are 1-hit, 2-hit, or multi-hit with respect to answer content tokens, comparing Naive RAG, RAG with pruning, and xMemory. Higher 2-hit and multi-hit indicate more evidence-dense retrieval under xMemory.
• Figure 5: Structural plasticity vs. downstream QA. We report Avg. BLEU/F1 (left axis) and the dynamic reassignment ratio (right axis) for four construction settings. Freezing high-level restructuring (w/o merge&split) yields 0% reassignment and lower accuracy, while the full system performs substantial reassignment (44.91%) and achieves the best Avg. BLEU/F1.

Theorems & Definitions (4)

• Theorem 2.1
• proof
• Corollary 2.2: Admissible candidate set size under bounded discriminability
• proof