From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding

TL;DR

This survey analyzes the progression of multimodal LLMs from image‑level capabilities to comprehensive long‑video understanding, highlighting how long videos introduce unique challenges such as between‑event and long‑term temporal reasoning. It reviews advances in model architecture, emphasizing long‑context backbones, efficient cross‑modal connectors, and time‑aware components, as well as training strategies that combine image/short/video pre‑training with long‑video‑specific data and instruction‑tuning. The paper provides a structured evaluation across benchmarks from seconds to hours, showing that long‑video specialized designs improve performance and that longer content remains challenging. It outlines future directions, including data resources, richer benchmarks, scalable frameworks, and broader application scenarios, to drive progress in long video understanding with LLMs.

Abstract

The integration of Large Language Models (LLMs) with visual encoders has recently shown promising performance in visual understanding tasks, leveraging their inherent capability to comprehend and generate human-like text for visual reasoning. Given the diverse nature of visual data, MultiModal Large Language Models (MM-LLMs) exhibit variations in model designing and training for understanding images, short videos, and long videos. Our paper focuses on the substantial differences and unique challenges posed by long video understanding compared to static image and short video understanding. Unlike static images, short videos encompass sequential frames with both spatial and within-event temporal information, while long videos consist of multiple events with between-event and long-term temporal information. In this survey, we aim to trace and summarize the advancements of MM-LLMs from image understanding to long video understanding. We review the differences among various visual understanding tasks and highlight the challenges in long video understanding, including more fine-grained spatiotemporal details, dynamic events, and long-term dependencies. We then provide a detailed summary of the advancements in MM-LLMs in terms of model design and training methodologies for understanding long videos. Finally, we compare the performance of existing MM-LLMs on video understanding benchmarks of various lengths and discuss potential future directions for MM-LLMs in long video understanding.

Figures (5)

• Figure 1: The development of MM-LMMs for multiple images, short videos and long videos.
• Figure 2: The comparison of MM-LLMs among Image-, Short-Video-, and Long-Video-LLMs. The bold content often highlights special considerations of LV-LLMs for long video understanding.
• Figure 3: Visual understanding of (a) images, (b) short videos, and (c) long videos.
• Figure 4: MM-LLMs of (b): Image-LLM, (c) Short-Video-LLM and (c) Long-Video-LLM.
• Figure 5: Long video sample for pretraining and instruction-tuning.