VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models

TL;DR

VisMem addresses the visual processing bottleneck in Vision-Language Models by introducing dual latent vision memories—short-term visually-dominant memory and long-term semantically-dominant memory—invoked on demand during autoregressive generation. The framework integrates a lightweight memory-formation path (memory query builder and memory formers) and a memory-invocation mechanism that inserts latent tokens into the generation stream without disrupting core capabilities. A two-stage reinforcement-learning training (Stage I memory formation, Stage II memory invocation) optimizes both components to maximize task performance while penalizing incorrect memory use. Across 12 benchmarks spanning understanding, reasoning, and generation, VisMem yields significant gains, demonstrates cross-domain transfer and continual-learning resilience, and remains efficient across diverse base models, signaling broad practical impact for scalable, memory-augmented VLMs.

Abstract

Despite the remarkable success of Vision-Language Models (VLMs), their performance on a range of complex visual tasks is often hindered by a "visual processing bottleneck": a propensity to lose grounding in visual evidence and exhibit a deficit in contextualized visual experience during prolonged generation. Drawing inspiration from human cognitive memory theory, which distinguishes short-term visually-dominant memory and long-term semantically-dominant memory, we propose VisMem, a cognitively-aligned framework that equips VLMs with dynamic latent vision memories, a short-term module for fine-grained perceptual retention and a long-term module for abstract semantic consolidation. These memories are seamlessly invoked during inference, allowing VLMs to maintain both perceptual fidelity and semantic consistency across thinking and generation. Extensive experiments across diverse visual benchmarks for understanding, reasoning, and generation reveal that VisMem delivers a significant average performance boost of 11.8% relative to the vanilla model and outperforms all counterparts, establishing a new paradigm for latent-space memory enhancement. The code will be available: https://github.com/YU-deep/VisMem.git.

Figures (10)

• Figure 1: Four primary paradigms for enhancing visual capabilities: (a) the direct training paradigm, (b) the image-level paradigm, (c) the token-level paradigm, and (d) the latent space paradigm. Our VisMem belongs to the last one, featuring latent vision memory.
• Figure 2: The overview of our proposed VisMem.
• Figure 3: Results of the cross-domain generalization study. Models are only trained on Visual CoT shao2024visual and Mulberry yao2024mulberry. Dashed bar indicates the results with full training data.
• Figure 4: Results of four-stage continual learning on MMVet yu2024mmvet. Stage 0 only includes itself, while stage 1, 2, 3 sequentially train models on different additional training data combinations.
• Figure 5: Results of memory invocation ratio and invocation relative position across four benchmarks.