The AI Hippocampus: How Far are We From Human Memory?

TL;DR

This survey surveys memory in LLMs and MLLMs across three paradigms—implicit (parametric), explicit (external memory), and agentic (persistent, task-oriented memory)—and extends the analysis to multimodal contexts. It consolidates methods for memorization, retrieval, and modification (including memory editing and unlearning), compares explicit memory representations (documents, graphs, and vectors), and reviews training strategies (pre-training and fine-tuning) that enable robust long-context and knowledge-intensive reasoning. The work also analyzes agent memory architectures (single- and multi-agent) with system-level design, evaluation metrics, and benchmarks, and surveys multimodal memory for audio, video, and other modalities in robotics and embodied AI, detailing architectures, tasks, and limitations. Overall, the paper highlights key challenges such as memory capacity, retrieval efficiency, alignment, hallucination, and cross-system interoperability, and points to future directions that emphasize deeper understanding of TransformerInternal mechanisms, scalable long-context solutions, dynamic memory adaptation, and cohesive multimodal memory systems.

Abstract

Memory plays a foundational role in augmenting the reasoning, adaptability, and contextual fidelity of modern Large Language Models and Multi-Modal LLMs. As these models transition from static predictors to interactive systems capable of continual learning and personalized inference, the incorporation of memory mechanisms has emerged as a central theme in their architectural and functional evolution. This survey presents a comprehensive and structured synthesis of memory in LLMs and MLLMs, organizing the literature into a cohesive taxonomy comprising implicit, explicit, and agentic memory paradigms. Specifically, the survey delineates three primary memory frameworks. Implicit memory refers to the knowledge embedded within the internal parameters of pre-trained transformers, encompassing their capacity for memorization, associative retrieval, and contextual reasoning. Recent work has explored methods to interpret, manipulate, and reconfigure this latent memory. Explicit memory involves external storage and retrieval components designed to augment model outputs with dynamic, queryable knowledge representations, such as textual corpora, dense vectors, and graph-based structures, thereby enabling scalable and updatable interaction with information sources. Agentic memory introduces persistent, temporally extended memory structures within autonomous agents, facilitating long-term planning, self-consistency, and collaborative behavior in multi-agent systems, with relevance to embodied and interactive AI. Extending beyond text, the survey examines the integration of memory within multi-modal settings, where coherence across vision, language, audio, and action modalities is essential. Key architectural advances, benchmark tasks, and open challenges are discussed, including issues related to memory capacity, alignment, factual consistency, and cross-system interoperability.

Figures (16)

• Figure 1: The overall framework of Memory Mechanisms in Large Language Models.
• Figure 2: Taxonomy of implicit memory in Transformer.
• Figure 3: Parameter vs. Circuit. The left graph demonstrates the location of memory stored in mlp layer and sa heads of the Transformer module, representing FNNs act as key-value memories, Different FFN neurons memorize different information, and manipulating attention head distributions, respectively. The right graph is a simplified demonstration of the knowledge flow between different transformer layers and various components within different layers.
• Figure 4: Three categories of Implicit Memory Modification
• Figure 5: Taxonomy of the structure design for learning with explicit memory.