Neuromem: A Granular Decomposition of the Streaming Lifecycle in External Memory for LLMs

TL;DR

Neuromem introduces a granular, streaming evaluation framework for External Memory Modules by decomposing the memory lifecycle into five design dimensions (D1–D5) and enforcing interleaved insertion and retrieval. The approach enables precise attribution of accuracy and latency to storage structure, normalization, consolidation, query formulation, and context integration, revealing that the storage data structure largely limits performance while aggressive preprocessing and generative retrieval incur a latency tax. Across LoCoMo, LONGMEMEVAL, and MemoryAgentBench, heuristic maintenance strategies offer superior real-time efficiency compared with costly generative approaches, which provide limited accuracy gains. Neuromem provides a reusable, open-source testbed for future work to optimize the full insert–maintain–retrieve–integrate loop under long-horizon entropy, guiding practical, energy-efficient deployments of memory-augmented LLMs.

Abstract

Most evaluations of External Memory Module assume a static setting: memory is built offline and queried at a fixed state. In practice, memory is streaming: new facts arrive continuously, insertions interleave with retrievals, and the memory state evolves while the model is serving queries. In this regime, accuracy and cost are governed by the full memory lifecycle, which encompasses the ingestion, maintenance, retrieval, and integration of information into generation. We present Neuromem, a scalable testbed that benchmarks External Memory Modules under an interleaved insertion-and-retrieval protocol and decomposes its lifecycle into five dimensions including memory data structure, normalization strategy, consolidation policy, query formulation strategy, and context integration mechanism. Using three representative datasets LOCOMO, LONGMEMEVAL, and MEMORYAGENTBENCH, Neuromem evaluates interchangeable variants within a shared serving stack, reporting token-level F1 and insertion/retrieval latency. Overall, we observe that performance typically degrades as memory grows across rounds, and time-related queries remain the most challenging category. The memory data structure largely determines the attainable quality frontier, while aggressive compression and generative integration mechanisms mostly shift cost between insertion and retrieval with limited accuracy gain.

Figures (8)

• Figure 1: Overview of Neuromem. We decompose the lifecycle into five dimensions (D1--D5) anchored by the (D1) data structure. The workflow spans the (b) Insertion Pipeline (D2 normalization, D3 consolidation) and (c) Retrieval Pipeline (D4 query formulation, D5 context integration), enabling controlled ablations under an interleaved insertion and retrieval protocol.
• Figure 2: Intrinsic Efficiency Frontier. Plotting accuracy against Intrinsic Structure Latency. The Inverted+Vector anchors the Pareto frontier on reasoning benchmarks (Left, Center), whereas the trade-off inverts on maintenance-heavy tasks (Right) where lightweight queues achieve superior efficiency.
• Figure 3: Breakdown of Cost-Effectiveness Evolution by Memory Lifecycle Dimension. The plots illustrate the F1-per-second trajectory across five rounds for varying strategies in D2--D5.
• Figure 4: Overview of the Neuromem Streaming Pipeline Architecture. The central PipelineCaller holds the test dataset and orchestrates the interleaved execution. It employs a backpressure mechanism to block the historical data stream during memory maintenance or reasoning evaluation, ensuring strict temporal causality.
• Figure 5: Comparison of Evaluation Protocols. Unlike the traditional Offline Protocol which queries a fixed memory state constructed in a single batch, our Interleaved Protocol performs retrieval at evolving memory states ($State 1, State 2, \dots$), enforcing strict temporal causality.