Chain-of-Memory: Lightweight Memory Construction with Dynamic Evolution for LLM Agents

TL;DR

The paper tackles the challenge of enabling LLM-based agents to maintain long-term memory under tight context windows by proposing Chain-of-Memory (CoM), a framework that replaces costly memory construction with lightweight retrieval and dynamic organization. CoM constructs memory as a flat, per-turn set of memory nodes and uses Top-$K$ retrieval followed by Dynamic Memory Chain Evolution, which builds coherent inference paths via a state-aware gating mechanism and adaptive truncation to prune noise. Empirically, CoM achieves notable accuracy gains (7.5%–10.4% absolute) on LongMemEval and LoCoMo while drastically reducing computation (about 2.7% of token usage and 6.0% of runtime) relative to heavy memory architectures, facilitating efficient long-horizon reasoning. The approach establishes a practical baseline for memory-enabled agents by balancing reasoning quality and efficiency and opens avenues for extensions to multi-modal contexts and alternative retrieval strategies. Key elements include memory nodes m_{i,j} = (x, $\tau$, $\rho$, $e$), Top-$K$ retrieval, and the gating-based chain expansion with $S_{gate}(m) = \cos(m, q) \times \cos(m, C_z^{(t)})$ and adaptive termination when $s_t^* < \beta \cdot s_{t-1}$.$

Abstract

External memory systems are pivotal for enabling Large Language Model (LLM) agents to maintain persistent knowledge and perform long-horizon decision-making. Existing paradigms typically follow a two-stage process: computationally expensive memory construction (e.g., structuring data into graphs) followed by naive retrieval-augmented generation. However, our empirical analysis reveals two fundamental limitations: complex construction incurs high costs with marginal performance gains, and simple context concatenation fails to bridge the gap between retrieval recall and reasoning accuracy. To address these challenges, we propose CoM (Chain-of-Memory), a novel framework that advocates for a paradigm shift toward lightweight construction paired with sophisticated utilization. CoM introduces a Chain-of-Memory mechanism that organizes retrieved fragments into coherent inference paths through dynamic evolution, utilizing adaptive truncation to prune irrelevant noise. Extensive experiments on the LongMemEval and LoCoMo benchmarks demonstrate that CoM outperforms strong baselines with accuracy gains of 7.5%-10.4%, while drastically reducing computational overhead to approximately 2.7% of token consumption and 6.0% of latency compared to complex memory architectures.

Figures (5)

• Figure 1: Empirical limitations of existing paradigms. (a) Heavy-weight memory construction strategies fail to demonstrate cost-effectiveness. (b) Naive retrieval strategies exhibit a reasoning bottleneck, where retrieved evidence is not effectively utilized for answer generation.
• Figure 2: The overview architecture of CoM. The workflow consists of two stages: (1) Memory Construction and Retrieval, and (2) Dynamic Memory Chain Evolution.
• Figure 3: Ablation Study Results. We compare the performance of our full method against variants removing specific components (w/o Framework, w/o DMCE, w/o APT) on GPT-4o-mini (a) and Qwen3-32B (b). The metrics include Accuracy (Acc), Token consumption, and Runtime. Our method achieves the best trade-off between accuracy and efficiency.
• Figure 4: The impact of hyperparameter $k$ on model accuracy and computational cost (tokens).
• Figure 5: Distribution of error types on the LongMemEval dataset. Reasoning failure constitute the majority of errors compared to retrieval failure.