LightMem: Lightweight and Efficient Memory-Augmented Generation

TL;DR

LightMem presents a lightweight, three-stage memory architecture for LLMs inspired by human memory, consisting of a pre-compressing sensory memory, a topic-aware short-term memory, and a sleep-time offline long-term memory consolidation. By decoupling update operations from online inference and applying targeted compression, topic segmentation, and parallel offline updates, LightMem significantly reduces token usage and API calls while maintaining or improving QA accuracy on long-horizon tasks. Empirical results on LongMemEval and LoCoMo with GPT and Qwen backbones show substantial improvements in efficiency (up to 38x total token reductions and up to 55x fewer API calls) and accuracy gains (up to around 7–29% depending on backbone and dataset). The work demonstrates that hierarchical, memory-augmented generation with offline consolidation can dramatically improve practical deployment of memory-enabled LLM agents, with code to be released for reproducibility.

Abstract

Despite their remarkable capabilities, Large Language Models (LLMs) struggle to effectively leverage historical interaction information in dynamic and complex environments. Memory systems enable LLMs to move beyond stateless interactions by introducing persistent information storage, retrieval, and utilization mechanisms. However, existing memory systems often introduce substantial time and computational overhead. To this end, we introduce a new memory system called LightMem, which strikes a balance between the performance and efficiency of memory systems. Inspired by the Atkinson-Shiffrin model of human memory, LightMem organizes memory into three complementary stages. First, cognition-inspired sensory memory rapidly filters irrelevant information through lightweight compression and groups information according to their topics. Next, topic-aware short-term memory consolidates these topic-based groups, organizing and summarizing content for more structured access. Finally, long-term memory with sleep-time update employs an offline procedure that decouples consolidation from online inference. On LongMemEval and LoCoMo, using GPT and Qwen backbones, LightMem consistently surpasses strong baselines, improving QA accuracy by up to 7.7% / 29.3%, reducing total token usage by up to 38x / 20.9x and API calls by up to 30x / 55.5x, while purely online test-time costs are even lower, achieving up to 106x / 117x token reduction and 159x / 310x fewer API calls. The code is available at https://github.com/zjunlp/LightMem.

Figures (5)

• Figure 1: Comparison of previous works and LightMem.
• Figure 2: The LightMem architecture. LightMem consists of three modules: a) An efficient Sensory Memory Module, b) a topic aware STM Module, and c) an LTM module updated in sleep time.
• Figure 3: Analysis and Ablation Study of Key Modules. Fig.(a) depicts the QA accuracy when using prompts compressed at different ratios ($r$) as in-contexts to query the LLM directly. Fig.(b) compares the accuracy of different topic segmentation methods under these varying compression ratios. Fig.(c1) and Fig.(c2) present the ablation study for the topic segmentation module, evaluating its impact on both performance and efficiency for the GPT and Qwen models.
• Figure 4: Impact of the STM buffer threshold ($th$) on performance and efficiency across different compression ratios ($r$). Each radar chart represents a specific configuration of a model (GPT-4o-mini or Qwen3) and a fixed compression ratio. The axes measure six key metrics: Accuracy (ACC), token consumption (Input, Output, Total), API Calls, and Runtime. To facilitate comparison, all values are normalized for visualization on the chart.
• Figure 5: Example of Topic Segment Attention Matrix.