Implicit Graph, Explicit Retrieval: Towards Efficient and Interpretable Long-horizon Memory for Large Language Models

TL;DR

LatentGraphMem addresses the challenge of long-horizon QA by integrating latent graph memory with explicit, budgeted subgraph retrieval. It introduces a three-stage pipeline: Stage I constructs a streaming, capped explicit graph for supervision; Stage II learns latent edge embeddings and retrieves a compact subgraph under a fixed budget with a differentiable TopK pathway; Stage III jointly fine-tunes builder and retriever while keeping the reasoner frozen, exposing only the retrieved subgraph to the reasoner. Across three long-horizon benchmarks and multiple frozen-reasoner backbones, LatentGraphMem achieves stronger average accuracy than explicit-graph and latent-memory baselines while maintaining stable inference times and enabling scalable reasoning to larger models. The approach offers interpretability through explicit evidence exposure without sacrificing efficiency, making it suitable for robust, scalable long-horizon reasoning in real-world applications.

Abstract

Long-horizon applications increasingly require large language models (LLMs) to answer queries when relevant evidence is sparse and dispersed across very long contexts. Existing memory systems largely follow two paradigms: explicit structured memories offer interpretability but often become brittle under long-context overload, while latent memory mechanisms are efficient and stable yet difficult to inspect. We propose LatentGraphMem, a memory framework that combines implicit graph memory with explicit subgraph retrieval. LatentGraphMem stores a graph-structured memory in latent space for stability and efficiency, and exposes a task-specific subgraph retrieval interface that returns a compact symbolic subgraph under a fixed budget for downstream reasoning and human inspection. During training, an explicit graph view is materialized to interface with a frozen reasoner for question-answering supervision. At inference time, retrieval is performed in latent space and only the retrieved subgraph is externalized. Experiments on long-horizon benchmarks across multiple model scales show that LatentGraphMem consistently outperforms representative explicit-graph and latent-memory baselines, while enabling parameter-efficient adaptation and flexible scaling to larger reasoners without introducing large symbolic artifacts.

Figures (5)

• Figure 1: (a) Performance of existing memory paradigms under long-horizon QA. NoMem: reasoner-only; RAG: retrieval-augmented generation; Explicit: explicit structured memory systems (A-Mem, PREMem, THEANINE, Mem0); Implicit: latent memory (MemGen). (b) Explicit and implicit memory paradigms for long-horizon reasoning. Existing explicit structured memory systems are interpretable but brittle under long and dispersed evidence, while implicit memory is more stable but opaque. LatentGraphMem combines implicit graph memory with explicit subgraph exposure.
• Figure 2: Overview of LatentGraphMem. Given a long-horizon $x$ and query $q$, LatentGraphMem builds an implicit graph-structured memory from $x$, extracts a compact explicit subgraph as evidence conditioned on $q$, and answers with a frozen LLM reasoner using only the question and the extracted subgraph. The context is used only for memory construction, and the lower panel shows a single-sample walkthrough.
• Figure 3: Memory Structure Ablation
• Figure 4: Effect of graph capacity $M$ on answer accuracy across three QA datasets.
• Figure 5: Case Study: Both Explicit and Implicit Construction Baselines Fail, While Our Method Succeeds