FreshMem: Brain-Inspired Frequency-Space Hybrid Memory for Streaming Video Understanding

TL;DR

FreshMem tackles the challenge of online streaming video understanding for multimodal models by introducing a brain-inspired memory framework that balances short-term fidelity with long-term coherence. It couples Multi-scale Frequency Memory (MFM), which projects overflowing frames into frequency coefficients to preserve a global gist, with Space Thumbnail Memory (STM), which discretizes streams into episodic clusters and compresses them into dense space thumbnails. The approach is training-free and demonstrates substantial accuracy gains on multiple streaming benchmarks, while also generalizing to offline long- and short-video datasets. By grounding memory in biological principles and using nonuniform temporal representations, FreshMem offers an efficient, scalable path toward robust long-horizon video understanding in real-world applications.

Abstract

Transitioning Multimodal Large Language Models (MLLMs) from offline to online streaming video understanding is essential for continuous perception. However, existing methods lack flexible adaptivity, leading to irreversible detail loss and context fragmentation. To resolve this, we propose FreshMem, a Frequency-Space Hybrid Memory network inspired by the brain's logarithmic perception and memory consolidation. FreshMem reconciles short-term fidelity with long-term coherence through two synergistic modules: Multi-scale Frequency Memory (MFM), which projects overflowing frames into representative frequency coefficients, complemented by residual details to reconstruct a global historical "gist"; and Space Thumbnail Memory (STM), which discretizes the continuous stream into episodic clusters by employing an adaptive compression strategy to distill them into high-density space thumbnails. Extensive experiments show that FreshMem significantly boosts the Qwen2-VL baseline, yielding gains of 5.20%, 4.52%, and 2.34% on StreamingBench, OV-Bench, and OVO-Bench, respectively. As a training-free solution, FreshMem outperforms several fully fine-tuned methods, offering a highly efficient paradigm for long-horizon streaming video understanding.

Figures (17)

• Figure 1: Overview of FreshMem and its biological foundations. (Top-Left) Logarithmic Perception, where the brain maintains high fidelity for immediate stimuli while distilling the distant past into a semantic “gist”; (Top-Right) Memory Consolidation, where SWRs compress continuous experiences into key space thumbnails; (Bottom) FreshMem materializes these principles via two synergistic modules: MFM and STM. This hybrid design enables the model to effectively reconcile short-term fidelity with long-term coherence for streaming video understanding.
• Figure 2: Illustration of FreshMem. (1) Sliding window for immediate high-fidelity context; (2) MFM, which maintains a global history by frequency coefficients and salient residuals; and (3) STM, which performs online episode segmentation and adaptive compression to keep key spatial thumbnails. The concatenated memory features enable the LLM to answer long-horizon queries with high accuracy.
• Figure 3: Interpretability analysis of the STM module. The t-SNE visualization illustrates the effectiveness of our Memory Consolidation mechanism, showing how continuous video streams are discretized into coherent episodic clusters.
• Figure 4: Reconstructed video stream from frequency coefficients. The visualization demonstrates the information decay inherent in our MFM module.
• Figure 5: Impact of Salient Residual Tokens. We visualize the reconstruction of the top-10% residual tokens selected by the MFM.