Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SwiftMem: Fast Agentic Memory via Query-aware Indexing

Anxin Tian, Yiming Li, Xing Li, Hui-Ling Zhen, Lei Chen, Xianzhi Yu, Zhenhua Dong, Mingxuan Yuan

TL;DR

SwiftMem tackles the latency bottleneck in memory-augmented LLMs by introducing a query-aware memory system with three synergistic components: a Temporal Index for time-based retrieval, a Semantic DAG-Tag Index for content-aware routing, and an Embedding co-consolidation mechanism to curb memory fragmentation and improve cache locality. The framework provides sub-linear retrieval, with time complexities such as $O(\log N_{mem})$ for explicit temporal queries and $O(k \cdot (\log |V| + D_{max}))$ for semantic tag routing, plus $O(|V|)$ consolidation. Empirical results on LoCoMo and LongMemEval show about a 47x faster search than state-of-the-art baselines while maintaining competitive accuracy (LLM scores around 0.70) and sub-15 ms search latency, supporting practical deployment of memory-augmented agents in long-term conversations. These findings highlight the practical impact of query-aware indexing for scalable, real-time agentic memory systems.

Abstract

Agentic memory systems have become critical for enabling LLM agents to maintain long-term context and retrieve relevant information efficiently. However, existing memory frameworks suffer from a fundamental limitation: they perform exhaustive retrieval across the entire storage layer regardless of query characteristics. This brute-force approach creates severe latency bottlenecks as memory grows, hindering real-time agent interactions. We propose SwiftMem, a query-aware agentic memory system that achieves sub-linear retrieval through specialized indexing over temporal and semantic dimensions. Our temporal index enables logarithmic-time range queries for time-sensitive retrieval, while the semantic DAG-Tag index maps queries to relevant topics through hierarchical tag structures. To address memory fragmentation during growth, we introduce an embedding-tag co-consolidation mechanism that reorganizes storage based on semantic clusters to improve cache locality. Experiments on LoCoMo and LongMemEval benchmarks demonstrate that SwiftMem achieves 47$\times$ faster search compared to state-of-the-art baselines while maintaining competitive accuracy, enabling practical deployment of memory-augmented LLM agents.

SwiftMem: Fast Agentic Memory via Query-aware Indexing

TL;DR

SwiftMem tackles the latency bottleneck in memory-augmented LLMs by introducing a query-aware memory system with three synergistic components: a Temporal Index for time-based retrieval, a Semantic DAG-Tag Index for content-aware routing, and an Embedding co-consolidation mechanism to curb memory fragmentation and improve cache locality. The framework provides sub-linear retrieval, with time complexities such as for explicit temporal queries and for semantic tag routing, plus consolidation. Empirical results on LoCoMo and LongMemEval show about a 47x faster search than state-of-the-art baselines while maintaining competitive accuracy (LLM scores around 0.70) and sub-15 ms search latency, supporting practical deployment of memory-augmented agents in long-term conversations. These findings highlight the practical impact of query-aware indexing for scalable, real-time agentic memory systems.

Abstract

Agentic memory systems have become critical for enabling LLM agents to maintain long-term context and retrieve relevant information efficiently. However, existing memory frameworks suffer from a fundamental limitation: they perform exhaustive retrieval across the entire storage layer regardless of query characteristics. This brute-force approach creates severe latency bottlenecks as memory grows, hindering real-time agent interactions. We propose SwiftMem, a query-aware agentic memory system that achieves sub-linear retrieval through specialized indexing over temporal and semantic dimensions. Our temporal index enables logarithmic-time range queries for time-sensitive retrieval, while the semantic DAG-Tag index maps queries to relevant topics through hierarchical tag structures. To address memory fragmentation during growth, we introduce an embedding-tag co-consolidation mechanism that reorganizes storage based on semantic clusters to improve cache locality. Experiments on LoCoMo and LongMemEval benchmarks demonstrate that SwiftMem achieves 47 faster search compared to state-of-the-art baselines while maintaining competitive accuracy, enabling practical deployment of memory-augmented LLM agents.
Paper Structure (29 sections, 2 theorems, 5 equations, 6 figures, 2 tables)

This paper contains 29 sections, 2 theorems, 5 equations, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Content-as-Memory
  4. Trajectory-as-Memory
  5. Methodology
  6. Preliminaries
  7. Memory Structures
  8. Storage and Retrieval
  9. Semantic DAG-Tag Index
  10. LLM-based Tag Generation
  11. DAG-based Tag Index
  12. Query-Tag Router
  13. Temporal Query
  14. Query with Explicit Time Intervals
  15. Implicit Temporal Query and Beyond
  16. ...and 14 more sections

Key Result

Theorem 3.2

For any path $p = (v_1, ..., v_k)$ in $\mathcal{G}$, let $\mathcal{S}(v_i)$ denote the semantic specificity of node $v_i$. The specificity increases monotonically along the path:

Figures (6)

  • Figure 1: Architectural comparison between existing memory frameworks and SwiftMem. Traditional approaches traverse the entire storage layer for each query (left), while SwiftMem employs query-aware indexing to fetch only relevant memory subsets (right).
  • Figure 2: SwiftMem's end-to-end query workflow. User queries are first analyzed by the Query-Tag Router and Temporal Indicator to determine relevant search dimensions. The three-tier indexing architecture (Temporal Index, DAG-Tag Index, and Embedding Index) then retrieves only the relevant memory subset from the storage layer, avoiding exhaustive scanning.
  • Figure 3: A Semantic DAG-based Tag Example between Andrew and Audrey on LoCoMo maharana2024evaluating. It organizes memories where pets serves as the root topic, with child concepts representing semantic aspects (dogs, companionship) and their derived emotions (happiness, comfort), enabling queries to match either specific terms or abstract themes.
  • Figure 4: The Scalability Evaluation on LoCoMo Dataset
  • Figure 5: The Ablation Study on Temporal Indexing.
  • ...and 1 more figures

Theorems & Definitions (4)

  • Definition 3.1: DAG-Tag Node Structure
  • Theorem 3.2: Semantic Specificity Hierarchy
  • Theorem 3.3: DAG-Tag Query Complexity
  • Definition 3.4: Temporal Index