All-Mem: Agentic Lifelong Memory via Dynamic Topology Evolution

Abstract

Lifelong interactive agents are expected to assist users over months or years, which requires continually writing long term memories while retrieving the right evidence for each new query under fixed context and latency budgets. Existing memory systems often degrade as histories grow, yielding redundant, outdated, or noisy retrieved contexts. We present All-Mem, an online/offline lifelong memory framework that maintains a topology structured memory bank via explicit, non destructive consolidation, avoiding the irreversible information loss typical of summarization based compression. In online operation, it anchors retrieval on a bounded visible surface to keep coarse search cost bounded. Periodically offline, an LLM diagnoser proposes confidence scored topology edits executed with gating using three operators: SPLIT, MERGE, and UPDATE, while preserving immutable evidence for traceability. At query time, typed links enable hop bounded, budgeted expansion from active anchors to archived evidence when needed. Experiments on LOCOMO and LONGMEMEVAL show improved retrieval and QA over representative baselines.

Figures (13)

• Figure 1: Comparison of prior works and All-Mem along real-time constraints, structural drift, and safe maintenance (traceability).
• Figure 2: All-Mem framework. (a) Online/offline decoupling with id-level buffering. (b) ATC performs parallel diagnosis with confidence-gating and non-destructive topology edits (Split$\rightarrow$Merge$\rightarrow$Update) while preserving versioned traceability. (c) Topology-aware retrieval anchors on the visible surface, performs budgeted typed-link expansion, and selects final memories for the agent.
• Figure 3: Accuracy--tokens sweep on LongMemEval-s. Each point corresponds to a retrieval budget setting: All-Mem and A-Mem vary $(K,k,L)$, while Mem0 varies $(K,k)$. The x-axis reports realized generator input tokens (all prompt tokens; mean over queries; excluding output tokens), and the y-axis reports F1.
• Figure 4: Stage-1 Top-$k$ similarity retrieval latency vs. history length $N$. We compare cosine-similarity retrieval over cached embeddings for the full memory bank $\mathcal{V}_N$ and the visible surface $\mathcal{V}_N^{+}$ at consolidation checkpoints. Shaded regions denote interquartile ranges (p25--p75). Searching over $\mathcal{V}_N^{+}$ yields consistently lower Stage-1 latency as $N$ increases.
• Figure 5: Budgeted recoverability analysis. Coverage $\mathrm{Cov}(H)$ of archived units reachable within hop budget $H$ over $\mathcal{E}_N$. The dashed line marks the hop budget achieving 95% coverage ($H_{0.95}=4$); the median distance is $H=2$.