Exploring the Role of Explicit Temporal Modeling in Multimodal Large Language Models for Video Understanding

TL;DR

This work addresses whether explicit temporal modeling is necessary for video understanding in Multimodal Large Language Models and introduces the Stackable Temporal Encoder (STE) as a flexible, convolution-based module. By embedding STE into open-source LLaVA-based backbones, the authors perform systematic comparisons between explicit and implicit temporal modeling across six video benchmarks and various frame-compression scenarios, showing consistent performance gains from explicit modeling and notable compression benefits. The study also delves into design factors such as temporal receptive fields and learning space, and demonstrates STE’s viability as a plug-in module with cross-modal implications, including for image modalities. Overall, the findings underscore the value of explicit temporal modeling in video MLLMs and provide practical guidance for future architecture design and deployment.

Abstract

Applying Multimodal Large Language Models (MLLMs) to video understanding presents significant challenges due to the need to model temporal relations across frames. Existing approaches adopt either implicit temporal modeling, relying solely on the LLM decoder, or explicit temporal modeling, employing auxiliary temporal encoders. To investigate this debate between the two paradigms, we propose the Stackable Temporal Encoder (STE). STE enables flexible explicit temporal modeling with adjustable temporal receptive fields and token compression ratios. Using STE, we systematically compare implicit and explicit temporal modeling across dimensions such as overall performance, token compression effectiveness, and temporal-specific understanding. We also explore STE's design considerations and broader impacts as a plug-in module and in image modalities. Our findings emphasize the critical role of explicit temporal modeling, providing actionable insights to advance video MLLMs.

Figures (6)

• Figure 1: (Left): Explicit temporal modeling may enhance temporal understanding compared to implicit temporal modeling. (Right): Performance on temporal-related tasks of LLaVA-OV with (labeled as STE) or without explicit temporal modeling across six benchmarks (arc colors indicate different benchmarks).
• Figure 2: (Left) Overview of our model for processing video inputs. (Right) Schematic diagram of the temporal encoder, comprising 2-layer STE modules that encode every four frames into one abstract frame through stacking two layers of 50% frame compression. The video, with dynamic length, is divided into convolutional units, and the STE is designed to handle diverse Input/Output (I/O) frame ratios based on these units. $T_{u,l}$, $T_{o,l}$, $T_{w,l}$, and $T_{s,l}$ denote the input frame count, the target output frame count, the convolutional window size, and the convolutional stride for a convolutional unit in the $l$-th layer, respectively.
• Figure 3: Performance when varying frame compressions: sampling frequency reduction vs. frame compression (STE), showing accuracy differences relative to backbones with 32 input frames.
• Figure 4: Performance of LLaVA-Video on temporal-related tasks equipped with (labeled as STE) or without explicit temporal modeling across benchmarks (arc colors indicate different benchmarks).
• Figure 5: Task-level performance on benchmarks. LLaVA-OV-STE and LLaVA-Video-STE refer to LLaVA-OV-STE-3-(2:2) and LLaVA-Video-STE-3-(2:2), respectively.)