Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialogues

Yaning Pan, Zekun Wang, Qianqian Xie, Yongqian Wen, Yuanxing Zhang, Guohui Zhang, Haoxuan Hu, Zhiyu Pan, Yibing Huang, Zhidong Gan, Yonghong Lin, An Ping, Tianhao Peng, Jiaheng Liu

TL;DR

MT-Video-Bench presents a holistic benchmark for evaluating multimodal LLMs in multi-turn, video-grounded dialogues, explicitly targeting six capabilities that blend perceptivity and interactivity. It comprises 987 dialogues over 135 videos across diverse domains, emphasizing cross-scene reasoning and long-range dependencies. Through extensive experiments on 20 models, the benchmark reveals substantial performance gaps, especially in interactive tasks and cross-scene settings, and highlights how context quality and prompt design influence results. The work offers a rigorous evaluation framework with practical implications for developing MLLMs capable of sustained, context-aware video conversations in real-world scenarios.

Abstract

The recent development of Multimodal Large Language Models (MLLMs) has significantly advanced AI's ability to understand visual modalities. However, existing evaluation benchmarks remain limited to single-turn question answering, overlooking the complexity of multi-turn dialogues in real-world scenarios. To bridge this gap, we introduce MT-Video-Bench, a holistic video understanding benchmark for evaluating MLLMs in multi-turn dialogues. Specifically, our MT-Video-Bench mainly assesses six core competencies that focus on perceptivity and interactivity, encompassing 987 meticulously curated multi-turn dialogues from diverse domains. These capabilities are rigorously aligned with real-world applications, such as interactive sports analysis and multi-turn video-based intelligent tutoring. With MT-Video-Bench, we extensively evaluate various state-of-the-art open-source and closed-source MLLMs, revealing their significant performance discrepancies and limitations in handling multi-turn video dialogues. The benchmark will be publicly available to foster future research.

MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialogues

TL;DR

MT-Video-Bench presents a holistic benchmark for evaluating multimodal LLMs in multi-turn, video-grounded dialogues, explicitly targeting six capabilities that blend perceptivity and interactivity. It comprises 987 dialogues over 135 videos across diverse domains, emphasizing cross-scene reasoning and long-range dependencies. Through extensive experiments on 20 models, the benchmark reveals substantial performance gaps, especially in interactive tasks and cross-scene settings, and highlights how context quality and prompt design influence results. The work offers a rigorous evaluation framework with practical implications for developing MLLMs capable of sustained, context-aware video conversations in real-world scenarios.

Abstract

The recent development of Multimodal Large Language Models (MLLMs) has significantly advanced AI's ability to understand visual modalities. However, existing evaluation benchmarks remain limited to single-turn question answering, overlooking the complexity of multi-turn dialogues in real-world scenarios. To bridge this gap, we introduce MT-Video-Bench, a holistic video understanding benchmark for evaluating MLLMs in multi-turn dialogues. Specifically, our MT-Video-Bench mainly assesses six core competencies that focus on perceptivity and interactivity, encompassing 987 meticulously curated multi-turn dialogues from diverse domains. These capabilities are rigorously aligned with real-world applications, such as interactive sports analysis and multi-turn video-based intelligent tutoring. With MT-Video-Bench, we extensively evaluate various state-of-the-art open-source and closed-source MLLMs, revealing their significant performance discrepancies and limitations in handling multi-turn video dialogues. The benchmark will be publicly available to foster future research.
Paper Structure (28 sections, 7 figures, 2 tables)

This paper contains 28 sections, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. MT-Video-Bench
  4. Overview
  5. Evaluation Tasks
  6. Data Collection
  7. Quality Control
  8. Dataset Statistics
  9. Evaluation Method
  10. Experiments
  11. Experimental Settings
  12. Main Results
  13. Further Analysis
  14. Performance comparison between single scene and cross scene
  15. Performance of different video lengths
  16. ...and 13 more sections

Figures (7)

  • Figure 1: Illustration of multi-turn dialogues under single-scene and cross-scene settings. The evaluated questions corresponding to tasks are marked with underlining, and the scenes involved in the entire multi-turn dialogues are marked with blue dotted boxes.
  • Figure 2: An overview of the semi-automatic data construction process of MT-Video-Bench.
  • Figure 3: Overview of MT-Video-Bench. (a) Video Categories. MT-Video-Bench includes videos spanning 5 major categories, ensuring diverse topical coverage. (b) Task Distribution. MT-Video-Bench consists of a total of 6 tasks with a relatively balanced distribution. (c) Video Duration Distribution. MT-Video-Bench includes both long and short videos. (d) Dialogue Turn Distribution. Multi-turn dialogues in MT-Video-Bench involve 5 to 8 rounds.
  • Figure 4: Performance comparison of Qwen2.5-VL-7B, InternVL3.5-8B (Think), and Gemini 2.5 Pro across various tasks under single-scene and cross-scene settings.
  • Figure 5: Performance of different video lengths, dialogue turns, and settings of dialogue context.
  • ...and 2 more figures