Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dual-Signal Adaptive KV-Cache Optimization for Long-Form Video Understanding in Vision-Language Models

Vishnu Sai, Dheeraj Sai, Srinath B, Girish Varma, Priyesh Shukla

TL;DR

Sali-Cache is proposed, a novel a priori optimization framework that implements dual-signal adaptive caching through proactive memory management that preserves context-rich features over extended temporal durations without degrading model performance, enabling efficient processing of long-form video content on consumer-grade hardware.

Abstract

Vision-Language Models (VLMs) face a critical memory bottleneck when processing long-form video content due to the linear growth of the Key-Value (KV) cache with sequence length. Existing solutions predominantly employ reactive eviction strategies that compute full attention matrices before discarding tokens, resulting in substantial computational waste. We propose Sali-Cache, a novel a priori optimization framework that implements dual-signal adaptive caching through proactive memory management. By integrating a temporal filter based on optical flow analysis for detecting inter-frame redundancy and a spatial filter leveraging saliency detection for identifying visually significant regions, Sali-Cache intelligently manages memory allocation before entering computationally expensive attention operations. Experimental evaluation on the LLaVA 1.6 architecture demonstrates that our method achieves a 2.20x compression ratio in effective memory usage while maintaining 100% accuracy across BLEU, ROUGE-L, and Exact Match metrics. Furthermore, under identical memory budget constraints, Sali-Cache preserves context-rich features over extended temporal durations without degrading model performance, enabling efficient processing of long-form video content on consumer-grade hardware.

Dual-Signal Adaptive KV-Cache Optimization for Long-Form Video Understanding in Vision-Language Models

TL;DR

Sali-Cache is proposed, a novel a priori optimization framework that implements dual-signal adaptive caching through proactive memory management that preserves context-rich features over extended temporal durations without degrading model performance, enabling efficient processing of long-form video content on consumer-grade hardware.

Abstract

Vision-Language Models (VLMs) face a critical memory bottleneck when processing long-form video content due to the linear growth of the Key-Value (KV) cache with sequence length. Existing solutions predominantly employ reactive eviction strategies that compute full attention matrices before discarding tokens, resulting in substantial computational waste. We propose Sali-Cache, a novel a priori optimization framework that implements dual-signal adaptive caching through proactive memory management. By integrating a temporal filter based on optical flow analysis for detecting inter-frame redundancy and a spatial filter leveraging saliency detection for identifying visually significant regions, Sali-Cache intelligently manages memory allocation before entering computationally expensive attention operations. Experimental evaluation on the LLaVA 1.6 architecture demonstrates that our method achieves a 2.20x compression ratio in effective memory usage while maintaining 100% accuracy across BLEU, ROUGE-L, and Exact Match metrics. Furthermore, under identical memory budget constraints, Sali-Cache preserves context-rich features over extended temporal durations without degrading model performance, enabling efficient processing of long-form video content on consumer-grade hardware.
Paper Structure (28 sections, 8 equations, 3 figures, 3 tables, 1 algorithm)

This paper contains 28 sections, 8 equations, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Vision-Language Model Architecture
  4. Visual Encoding Pipeline
  5. Adapter Projection
  6. The KV-Cache Memory Challenge
  7. Existing KV-Cache Optimization Methods
  8. Reactive Eviction Strategies
  9. Sliding Window Approaches
  10. Quantization-Based Methods
  11. Methodology
  12. Temporal Filtering via Optical Flow
  13. Spatial Filtering via Saliency-Guided Quantization
  14. Just-In-Time Dequantization
  15. Experimental Evaluation
  16. ...and 13 more sections

Figures (3)

  • Figure 1: Sali-Cache architecture overview. The dual-signal pipeline applies temporal filtering via optical flow analysis followed by spatial filtering through saliency-guided quantization before KV-cache storage. Frames classified as temporally redundant bypass computation entirely through cache reuse, while non-redundant frames undergo spatial analysis to determine per-patch quantization levels.
  • Figure 2: Patch-level compression breakdown across 100 frames. The stacked bars show the distribution of patches across different processing categories: skipped (cache reuse), pruned, INT4, INT8, and FP16.
  • Figure 3: Cumulative memory savings over frame sequence. The growing gap between computed and skipped patches demonstrates increasing benefits for longer videos.