Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Empirical Study for Representations of Videos in Video Question Answering via MLLMs

Zhi Li, Yanan Wang, Hao Niu, Julio Vizcarra, Masato Taya

TL;DR

This paper tackles identifying effective video representations for VideoQA using multimodal large language models. It introduces a two-stage framework: a representation extraction module that encodes visual frames, audio signals, and subtitles, and a QA module that queries frozen LLMs with multimodal embeddings. The authors conduct an extensive empirical study across single- and multi-modality setups on VideoMME and LongVideoBench, evaluating accuracy, GPU VRAM usage, and inference latency. Key findings show that visual frames deliver the largest accuracy gains but incur substantial memory and time costs, while subtitles provide a compact yet informative signal, and the combination Q+S+V yields the best overall performance when resources permit. These insights guide the design of resource-aware VideoQA systems and help practitioners choose representations based on budget and latency constraints.

Abstract

Multimodal large language models have recently achieved remarkable progress in video question answering (VideoQA) by jointly processing visual, textual, and audio information. However, it remains unclear which video representations are most effective for MLLMs, and how different modalities balance task accuracy against computational efficiency. In this work, we present a comprehensive empirical study of video representation methods for VideoQA with MLLMs. We systematically evaluate single modality inputs question only, subtitles, visual frames, and audio signals as well as multimodal combinations, on two widely used benchmarks: VideoMME and LongVideoBench. Our results show that visual frames substantially enhance accuracy but impose heavy costs in GPU memory and inference latency, while subtitles provide a lightweight yet effective alternative, particularly for long videos. These findings highlight clear trade-offs between effectiveness and efficiency and provide practical insights for designing resource-aware MLLM-based VideoQA systems.

An Empirical Study for Representations of Videos in Video Question Answering via MLLMs

TL;DR

This paper tackles identifying effective video representations for VideoQA using multimodal large language models. It introduces a two-stage framework: a representation extraction module that encodes visual frames, audio signals, and subtitles, and a QA module that queries frozen LLMs with multimodal embeddings. The authors conduct an extensive empirical study across single- and multi-modality setups on VideoMME and LongVideoBench, evaluating accuracy, GPU VRAM usage, and inference latency. Key findings show that visual frames deliver the largest accuracy gains but incur substantial memory and time costs, while subtitles provide a compact yet informative signal, and the combination Q+S+V yields the best overall performance when resources permit. These insights guide the design of resource-aware VideoQA systems and help practitioners choose representations based on budget and latency constraints.

Abstract

Multimodal large language models have recently achieved remarkable progress in video question answering (VideoQA) by jointly processing visual, textual, and audio information. However, it remains unclear which video representations are most effective for MLLMs, and how different modalities balance task accuracy against computational efficiency. In this work, we present a comprehensive empirical study of video representation methods for VideoQA with MLLMs. We systematically evaluate single modality inputs question only, subtitles, visual frames, and audio signals as well as multimodal combinations, on two widely used benchmarks: VideoMME and LongVideoBench. Our results show that visual frames substantially enhance accuracy but impose heavy costs in GPU memory and inference latency, while subtitles provide a lightweight yet effective alternative, particularly for long videos. These findings highlight clear trade-offs between effectiveness and efficiency and provide practical insights for designing resource-aware MLLM-based VideoQA systems.
Paper Structure (8 sections, 6 equations, 3 figures, 1 table)

This paper contains 8 sections, 6 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Representation Extraction
  4. Video Question Answering
  5. Experiments
  6. Experimental Setup
  7. Experimental Result
  8. Conclusion

Figures (3)

  • Figure 1: An overview of our framework. It consists of a representation extraction module and a question answering module. The first extracts multimodal video representations and the second inputs those representations into a MLLM to answer questions.
  • Figure 2: GPU memory consumption across different input modalities on VideoMME.
  • Figure 3: Inference latency across different input modalities on VideoMME. We inference 7B and 72B models on 1 and 3 NVIDIA A100 GPUs, respectively