MMViR: A Multi-Modal and Multi-Granularity Representation for Long-range Video Understanding

TL;DR

MMViR tackles the inefficiency of processing hour-long videos with MLLMs by introducing a tri-level, multi-modal representation that combines a global timeline, coarse action-level summaries, and fine-grained frame-grounded descriptions. The construction pipeline segments videos at turning points, builds a timeline, and generates retrieval-friendly representations that can be selectively expanded for downstream tasks. Across five long-video benchmarks, MMViR achieves superior accuracy while reducing inference latency, demonstrating better efficiency-accuracy trade-offs than prior methods. This work provides a practical framework for scalable long-video understanding with broad applicability to QA, summarization, and retrieval.

Abstract

Long videos, ranging from minutes to hours, present significant challenges for current Multi-modal Large Language Models (MLLMs) due to their complex events, diverse scenes, and long-range dependencies. Direct encoding of such videos is computationally too expensive, while simple video-to-text conversion often results in redundant or fragmented content. To address these limitations, we introduce MMViR, a novel multi-modal, multi-grained structured representation for long video understanding. MMViR identifies key turning points to segment the video and constructs a three-level description that couples global narratives with fine-grained visual details. This design supports efficient query-based retrieval and generalizes well across various scenarios. Extensive evaluations across three tasks, including QA, summarization, and retrieval, show that MMViR outperforms the prior strongest method, achieving a 19.67% improvement in hour-long video understanding while reducing processing latency to 45.4% of the original.

Figures (13)

• Figure 1: Performance comparison on Hourvideo QA benchmark. MMViR consistently outperforms prior methods by simultaneously optimizing for reasoning precision and inference efficiency.
• Figure 2: Overall distribution of frame-to-frame CLIP Similarity Scores of a sampled hour-long video.
• Figure 3: Token cost vs. QA accuracy for different granularities of video representations.
• Figure 4: Token cost vs. QA accuracy for different modalities of video representations.
• Figure 5: Overview of the MMViR and the downstream inference. Given a long video, MMViR leverages an MLLM to construct a multi-modal and multi-grained representation for it. The high-level timeline descriptions serve as a global semantic index, enabling efficient query-aware relevant clip localization. This design ensures a synergistic balance between global narrative coherence and fine-grained evidence, effectively addressing the computational challenges of long-form content.