SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Models

TL;DR

SPORTU introduces a comprehensive, cross-sport benchmark for evaluating multimodal large language models on sports understanding, comprising SPORTU-text (900 explanations-enabled QA pairs across five sports) and SPORTU-video (1,701 slow-motion clips across seven sports with 12,048 QA pairs across Easy/Medium/Hard). The framework tests multi-level reasoning from basic sport recognition to foul detection and rule application, combining pure-text QA with slow-motion video QA. Empirical results show GPT-4o excels in text reasoning while video tasks lag behind, with large gaps in deep rule-based reasoning and domain knowledge; prompting strategies that force step-by-step reasoning often degrade performance, and automatic metrics like G-Eval only weakly align with human judgments. SPORTU highlights critical gaps in current MLLMs’ ability to connect observed actions to rules and generate faithful explanations, indicating clear directions for future improvement in grounding, reasoning, and evaluation.)

Abstract

Multimodal Large Language Models (MLLMs) are advancing the ability to reason about complex sports scenarios by integrating textual and visual information. To comprehensively evaluate their capabilities, we introduce SPORTU, a benchmark designed to assess MLLMs across multi-level sports reasoning tasks. SPORTU comprises two key components: SPORTU-text, featuring 900 multiple-choice questions with human-annotated explanations for rule comprehension and strategy understanding. This component focuses on testing models' ability to reason about sports solely through question-answering (QA), without requiring visual inputs; SPORTU-video, consisting of 1,701 slow-motion video clips across 7 different sports and 12,048 QA pairs, designed to assess multi-level reasoning, from simple sports recognition to complex tasks like foul detection and rule application. We evaluate four prevalent LLMs mainly utilizing few-shot learning paradigms supplemented by chain-of-thought (CoT) prompting on the SPORTU-text part. We evaluate four LLMs using few-shot learning and chain-of-thought (CoT) prompting on SPORTU-text. GPT-4o achieves the highest accuracy of 71%, but still falls short of human-level performance, highlighting room for improvement in rule comprehension and reasoning. The evaluation for the SPORTU-video part includes 7 proprietary and 6 open-source MLLMs. Experiments show that models fall short on hard tasks that require deep reasoning and rule-based understanding. Claude-3.5-Sonnet performs the best with only 52.6% accuracy on the hard task, showing large room for improvement. We hope that SPORTU will serve as a critical step toward evaluating models' capabilities in sports understanding and reasoning.

Figures (47)

• Figure 1: SPORTU consists of two parts: SPORTU-text, which evaluates sports understanding through text-based multiple-choice questions, and SPORTU-video, which assesses multi-level reasoning through video-based QA tasks. SPORTU provides a comprehensive sports understanding evaluation of multi-leveled reasoning beyond perception. Right side is a sample from the scenario-based question in SPORTU-text, along with examples of both hard-level and easy-level questions from SPORTU-video.
• Figure 2: Examples of SPORTU-text (left) and SPORTU-video (right). Figure \ref{['fig:sport_u']} also shows an example of an open-ended SPORTU-video question.
• Figure 3: Error type distribution across different MLLMs on SPORTU-video tasks. The analysis reveals that Question Understanding Error is the most common issue, followed by Hallucination Error. Each error type highlights specific model limitations in comprehending the task.
• Figure 4: Under camera angle one, the model answers the question correctly.
• Figure 5: Under camera angle two, the model answers the question wrong.