FiLA-Video: Spatio-Temporal Compression for Fine-Grained Long Video Understanding

TL;DR

FiLA-Video tackles long-video understanding in multimodal large language models by introducing a scene-aware spatio-temporal compression pipeline that selects important scenes via clustering and merges frames with a lightweight, learnable fusion module. It combines a synthetic long-video data strategy with a four-stage training schedule to boost temporal understanding while keeping token counts manageable. Empirical results show FiLA-Video achieves superior efficiency and accuracy across multiple benchmarks, outperforming baselines with fewer visual tokens and demonstrating robustness through ablations. The work provides practical impact by enabling scalable long-video reasoning and releasing an open-source pipeline and long-video captioning dataset for broader use.

Abstract

Recent advancements in video understanding within visual large language models (VLLMs) have led to notable progress. However, the complexity of video data and contextual processing limitations still hinder long-video comprehension. A common approach is video feature compression to reduce token input to large language models, yet many methods either fail to prioritize essential features, leading to redundant inter-frame information, or introduce computationally expensive modules.To address these issues, we propose FiLA(Fine-grained Vision Language Model)-Video, a novel framework that leverages a lightweight dynamic-weight multi-frame fusion strategy, which adaptively integrates multiple frames into a single representation while preserving key video information and reducing computational costs. To enhance frame selection for fusion, we introduce a keyframe selection strategy, effectively identifying informative frames from a larger pool for improved summarization. Additionally, we present a simple yet effective long-video training data generation strategy, boosting model performance without extensive manual annotation. Experimental results demonstrate that FiLA-Video achieves superior efficiency and accuracy in long-video comprehension compared to existing methods.

Figures (7)

• Figure 1: Comparison between the frames selected by Uniformly Sampling and Scene-Selection respectively. We show the first 8 frames collected from the same video. The left pair shows that the method of uniformly sampling frames inevitably collects too many repeated frames, which leads to a waste of computing resources during token compression. The pair on the right solves the problem.
• Figure 1: Qualitative results on video QA and video caption task. For each video among these cases, we sample 96 frames.
• Figure 2: The pipeline for our synthetic data.
• Figure 3: (Left)Visualization of the video duration. (Right)Visualization of the video caption length.
• Figure 4: Different merging methods.