Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TennisTV: Do Multimodal Large Language Models Understand Tennis Rallies?

Zhongyuan Bao, Lejun Zhang

TL;DR

TennisTV introduces a domain-specific benchmark for tennis video understanding by modeling each rally as a time-ordered sequence of stroke events and defining eight tasks that span stroke-level perception to rally-level reasoning. It employs an automated annotation workflow to generate 2527 questions across 1298 videos, enabling systematic evaluation of 17 multimodal LLMs (both open- and closed-source) on fast-paced, information-dense rallies. Key findings show strong correlations among low-level tasks (AR, TI, TP), a task-dependent optimal frame-sampling density, and that explicit reasoning helps mainly knowledge-light tasks while temporal grounding remains a bottleneck. The work provides practical takeaways to balance sampling density and to enhance temporal grounding through reinforcement learning-based alignment, positioning TennisTV as a valuable benchmark to accelerate progress in dynamic sports video understanding.

Abstract

Multimodal large language models (MLLMs) excel at general video understanding but struggle with fast, high-frequency sports like tennis, where rally clips are short yet information-dense. To systematically evaluate MLLMs in this challenging domain, we present TennisTV, the first and most comprehensive benchmark for tennis video understanding. TennisTV models each rally as a temporal-ordered sequence of consecutive stroke events, using automated pipelines for filtering and question generation. It covers 8 tasks from the stroke level to the rally level and includes 2527 human-verified questions. Evaluating 17 representative MLLMs, we provide the first systematic assessment of tennis video understanding. Results yield two key insights: (i) frame-sampling density should be tailored and balanced across tasks, and (ii) improving temporal grounding is essential for stronger reasoning.

TennisTV: Do Multimodal Large Language Models Understand Tennis Rallies?

TL;DR

TennisTV introduces a domain-specific benchmark for tennis video understanding by modeling each rally as a time-ordered sequence of stroke events and defining eight tasks that span stroke-level perception to rally-level reasoning. It employs an automated annotation workflow to generate 2527 questions across 1298 videos, enabling systematic evaluation of 17 multimodal LLMs (both open- and closed-source) on fast-paced, information-dense rallies. Key findings show strong correlations among low-level tasks (AR, TI, TP), a task-dependent optimal frame-sampling density, and that explicit reasoning helps mainly knowledge-light tasks while temporal grounding remains a bottleneck. The work provides practical takeaways to balance sampling density and to enhance temporal grounding through reinforcement learning-based alignment, positioning TennisTV as a valuable benchmark to accelerate progress in dynamic sports video understanding.

Abstract

Multimodal large language models (MLLMs) excel at general video understanding but struggle with fast, high-frequency sports like tennis, where rally clips are short yet information-dense. To systematically evaluate MLLMs in this challenging domain, we present TennisTV, the first and most comprehensive benchmark for tennis video understanding. TennisTV models each rally as a temporal-ordered sequence of consecutive stroke events, using automated pipelines for filtering and question generation. It covers 8 tasks from the stroke level to the rally level and includes 2527 human-verified questions. Evaluating 17 representative MLLMs, we provide the first systematic assessment of tennis video understanding. Results yield two key insights: (i) frame-sampling density should be tailored and balanced across tasks, and (ii) improving temporal grounding is essential for stronger reasoning.

Paper Structure

This paper contains 13 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Sports Understanding
  4. MLLMs for Video
  5. TennisTV Benchmark
  6. Benchmark Task Analysis
  7. Dataset Annotation Pipeline
  8. Experiment
  9. Implementation Details
  10. Evaluation Metric
  11. Main Results
  12. Takeaways
  13. Conclusion

Figures (5)

  • Figure 1: Illustrations of a simple counting task easily handled by non-experts, but still challenging for the advanced commercial models.
  • Figure 2: Overview and annotation pipeline of TennisTV Benchmark.
  • Figure 3: Benchmark statistics of TennisTV.
  • Figure 4: Correlation matrix between 8 tasks in TennisTV.
  • Figure 5: Visualization of performance on three sub-tasks, which require no prior knowledge of tennis, based on the same base model.