Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring the Design Space of Visual Context Representation in Video MLLMs

Yifan Du, Yuqi Huo, Kun Zhou, Zijia Zhao, Haoyu Lu, Han Huang, Wayne Xin Zhao, Bingning Wang, Weipeng Chen, Ji-Rong Wen

TL;DR

This paper forms the task of visual context representation as a constrained optimization problem, and model the language modeling loss as a function of the number of frames and the number of embeddings per frame, given the maximum visual context window size.

Abstract

Video Multimodal Large Language Models (MLLMs) have shown remarkable capability of understanding the video semantics on various downstream tasks. Despite the advancements, there is still a lack of systematic research on visual context representation, which refers to the scheme to select frames from a video and further select the tokens from a frame. In this paper, we explore the design space for visual context representation, and aim to improve the performance of video MLLMs by finding more effective representation schemes. Firstly, we formulate the task of visual context representation as a constrained optimization problem, and model the language modeling loss as a function of the number of frames and the number of embeddings (or tokens) per frame, given the maximum visual context window size. Then, we explore the scaling effects in frame selection and token selection respectively, and fit the corresponding function curve by conducting extensive empirical experiments. We examine the effectiveness of typical selection strategies and present empirical findings to determine the two factors. Furthermore, we study the joint effect of frame selection and token selection, and derive the optimal formula for determining the two factors. We demonstrate that the derived optimal settings show alignment with the best-performed results of empirical experiments. Our code and model are available at: https://github.com/RUCAIBox/Opt-Visor.

Exploring the Design Space of Visual Context Representation in Video MLLMs

TL;DR

This paper forms the task of visual context representation as a constrained optimization problem, and model the language modeling loss as a function of the number of frames and the number of embeddings per frame, given the maximum visual context window size.

Abstract

Video Multimodal Large Language Models (MLLMs) have shown remarkable capability of understanding the video semantics on various downstream tasks. Despite the advancements, there is still a lack of systematic research on visual context representation, which refers to the scheme to select frames from a video and further select the tokens from a frame. In this paper, we explore the design space for visual context representation, and aim to improve the performance of video MLLMs by finding more effective representation schemes. Firstly, we formulate the task of visual context representation as a constrained optimization problem, and model the language modeling loss as a function of the number of frames and the number of embeddings (or tokens) per frame, given the maximum visual context window size. Then, we explore the scaling effects in frame selection and token selection respectively, and fit the corresponding function curve by conducting extensive empirical experiments. We examine the effectiveness of typical selection strategies and present empirical findings to determine the two factors. Furthermore, we study the joint effect of frame selection and token selection, and derive the optimal formula for determining the two factors. We demonstrate that the derived optimal settings show alignment with the best-performed results of empirical experiments. Our code and model are available at: https://github.com/RUCAIBox/Opt-Visor.

Paper Structure

This paper contains 37 sections, 7 equations, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Model Architecture.
  4. Training Data.
  5. Implementation Details.
  6. Evaluation Setup
  7. Scaling law of visual context
  8. Problem Formulation
  9. Scaling Effect of the Visual Embeddings
  10. Sampling-based Method
  11. Experimental Setup.
  12. Fitting Function.
  13. Benchmark Performance Analysis
  14. Compression-based Method
  15. Experimental Setup.
  16. ...and 22 more sections

Figures (4)

  • Figure 1: Overview of the LLaVA-like architecture for video-MLLM, and our used frame-level and embedding-level operations for adjusting the visual context window size.
  • Figure 2: The scaling law of visual embeddings, reflected by the language modeling loss and the zero-shot accuracy on video understanding benchmarks.
  • Figure 3: The scaling law of frames, reflected by the language modeling loss and the zero-shot accuracy on video understanding benchmarks.
  • Figure 4: The scaling law of the visual embeddings and frames. We also display the maximum point and the minimum point in our experiments.