Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LVC: A Lightweight Compression Framework for Enhancing VLMs in Long Video Understanding

Ziyi Wang, Haoran Wu, Yiming Rong, Deyang Jiang, Yixin Zhang, Yunlong Zhao, Shuang Xu, Bo XU

TL;DR

LVC addresses the challenge of long-video understanding by turning dense, temporally rich information into compact pseudo-image frames through a parameter-free, query-guided compression module. By training only the alignment layer, LVC equips existing VLMs with temporal awareness without heavy pretraining or LLM fine-tuning, achieving consistent gains on MLVU and Video-MME across multiple model scales. The approach reduces data and compute costs while preserving performance, and experiments show LVC can outperform some strong baselines on long-video benchmarks, signaling a practical path toward efficient temporal reasoning in vision-language systems.

Abstract

Long video understanding is a complex task that requires both spatial detail and temporal awareness. While Vision-Language Models (VLMs) obtain frame-level understanding capabilities through multi-frame input, they suffer from information loss due to the sparse sampling strategy. In contrast, Video Large Language Models (Video-LLMs) capture temporal relationships within visual features but are limited by the scarcity of high-quality video-text datasets. To transfer long video understanding capabilities to VLMs with minimal data and computational cost, we propose Lightweight Video Compression (LVC), a novel method featuring the Query-Attention Video Compression mechanism, which effectively tackles the sparse sampling problem in VLMs. By training only the alignment layer with 10k short video-text pairs, LVC significantly enhances the temporal reasoning abilities of VLMs. Extensive experiments show that LVC provides consistent performance improvements across various models, including the InternVL2 series and Phi-3.5-Vision. Notably, the InternVL2-40B-LVC achieves scores of 68.2 and 65.9 on the long video understanding benchmarks MLVU and Video-MME, respectively, with relative improvements of 14.6% and 7.7%. The enhanced models and code will be publicly available soon.

LVC: A Lightweight Compression Framework for Enhancing VLMs in Long Video Understanding

TL;DR

LVC addresses the challenge of long-video understanding by turning dense, temporally rich information into compact pseudo-image frames through a parameter-free, query-guided compression module. By training only the alignment layer, LVC equips existing VLMs with temporal awareness without heavy pretraining or LLM fine-tuning, achieving consistent gains on MLVU and Video-MME across multiple model scales. The approach reduces data and compute costs while preserving performance, and experiments show LVC can outperform some strong baselines on long-video benchmarks, signaling a practical path toward efficient temporal reasoning in vision-language systems.

Abstract

Long video understanding is a complex task that requires both spatial detail and temporal awareness. While Vision-Language Models (VLMs) obtain frame-level understanding capabilities through multi-frame input, they suffer from information loss due to the sparse sampling strategy. In contrast, Video Large Language Models (Video-LLMs) capture temporal relationships within visual features but are limited by the scarcity of high-quality video-text datasets. To transfer long video understanding capabilities to VLMs with minimal data and computational cost, we propose Lightweight Video Compression (LVC), a novel method featuring the Query-Attention Video Compression mechanism, which effectively tackles the sparse sampling problem in VLMs. By training only the alignment layer with 10k short video-text pairs, LVC significantly enhances the temporal reasoning abilities of VLMs. Extensive experiments show that LVC provides consistent performance improvements across various models, including the InternVL2 series and Phi-3.5-Vision. Notably, the InternVL2-40B-LVC achieves scores of 68.2 and 65.9 on the long video understanding benchmarks MLVU and Video-MME, respectively, with relative improvements of 14.6% and 7.7%. The enhanced models and code will be publicly available soon.

Paper Structure

This paper contains 18 sections, 4 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-Language Models
  4. Video Large Language Models
  5. Long Video Understanding
  6. Methods
  7. Problem Formulation of Video Understanding
  8. Model Architecture
  9. Infusing Queries into Video Compressing
  10. Experiments
  11. Dataset and Benchmarks
  12. Experimental Setup
  13. Results on InternVL2 Series
  14. Analyses on Input Frames
  15. Ablation Studies
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Comparison examples of InternVL2-40B and InternVL2-40B-LVC. Red text indicates errors in InternVL2-40B caused by its sparse sampling strategy failing to capture temporal details, while green text highlights corrections made by InternVL2-40B-LVC.
  • Figure 2: (a)VLMs (b)Video-LLMs (c)The overall architecture of LVC.
  • Figure 3: Performance comparison of InternVL2-8/26/40B before and after LVC enhancement on MLVU and Video-MME, with input frames set to 2/4/8/16. For LVC models, input frames refer to the numbers of pseudo-image frame after compression. Sampled frames are uniformly set to 64.
  • Figure 4: Performance of InternVL2-8 on MLVU with LVC enhancement at different training steps.
  • Figure 5: Performance of Phi-3.5-Vision before and after LVC enhancement, along with data ablation results (middle).