HiMem: Hierarchical Long-Term Memory for LLM Long-Horizon Agents

TL;DR

HiMem addresses the challenge of preserving and leveraging long-term memory in LLM agents for extended dialogues by proposing a hierarchical memory system that combines Episode Memory for fine-grained events with Note Memory for stable knowledge. It introduces a Topic-Aware Event--Surprise Dual-Channel Segmentation to build cognitively coherent episodes and a multi-stage knowledge extraction pipeline to form concise, query-friendly notes, all connected through selective Knowledge Alignment. Retrieval can proceed in a hybrid or best-effort fashion, with conflict-aware Memory Reconsolidation updating notes when retrieved evidence is insufficient, enabling continual self-evolution. Empirical results on the LoCoMo long-horizon benchmark show consistent improvements in accuracy, consistency, and open-domain reasoning, with ablations validating the necessity of the hierarchical memory structure, semantic alignment, and the self-evolution mechanism for robust long-term reasoning.

Abstract

Although long-term memory systems have made substantial progress in recent years, they still exhibit clear limitations in adaptability, scalability, and self-evolution under continuous interaction settings. Inspired by cognitive theories, we propose HiMem, a hierarchical long-term memory framework for long-horizon dialogues, designed to support memory construction, retrieval, and dynamic updating during sustained interactions. HiMem constructs cognitively consistent Episode Memory via a Topic-Aware Event--Surprise Dual-Channel Segmentation strategy, and builds Note Memory that captures stable knowledge through a multi-stage information extraction pipeline. These two memory types are semantically linked to form a hierarchical structure that bridges concrete interaction events and abstract knowledge, enabling efficient retrieval without sacrificing information fidelity. HiMem supports both hybrid and best-effort retrieval strategies to balance accuracy and efficiency, and incorporates conflict-aware Memory Reconsolidation to revise and supplement stored knowledge based on retrieval feedback. This design enables continual memory self-evolution over long-term use. Experimental results on long-horizon dialogue benchmarks demonstrate that HiMem consistently outperforms representative baselines in accuracy, consistency, and long-term reasoning, while maintaining favorable efficiency. Overall, HiMem provides a principled and scalable design paradigm for building adaptive and self-evolving LLM-based conversational agents. The code is available at https://github.com/jojopdq/HiMem.

Figures (4)

• Figure 1: Overview of HiMem. (A) Memory organization: a hierarchical connection between Episode Memory and Note Memory. (B) Memory construction: pipelines that transform dialogue logs into Episode Memory and Note Memory. (C) Best-effort retrieval: hierarchical retrieval in the order of Note Memory $\rightarrow$ Episode Memory, with an LLM assessing evidence sufficiency. (D) Memory self-evolution: when evidence from Note Memory is insufficient, the system supplements potentially missing information from Episode Memory and triggers conflict detection and updating.
• Figure 2: Performance comparison (GPT-Score) before and after enabling Memory Self-Evolution for Note Memory and HiMem. Memory Self-Evolution is triggered through conflict-aware Memory Reconsolidation during best-effort retrieval.
• Figure 3: Mapping of representative long-term memory systems under the Memory Form–Memory Organization–Memory Operation framework. The framework characterizes long-term memory systems along three dimensions: memory unit representation, organizational structure, and memory operations. When a dimension collapses into a single fixed design choice that restricts adaptive trade-offs among granularity, structure, or temporal evolution, it is considered to exhibit design degeneration. In contrast, HiMem maintains non-degenerate designs across all three dimensions, enabling more flexible and evolvable long-term memory modeling.
• Figure 4: Effect of top-k on performance and efficiency. We report GPT-Score, latency, and token consumption as a function of $k$. Latency reports search latency only.