Towards Universal Soccer Video Understanding

TL;DR

This work tackles comprehensive soccer video understanding by introducing SoccerReplay-1988, the largest multi-modal soccer dataset with automated curation, and MatchVision, a soccer-specific spatiotemporal visual encoder. The framework unifies event classification, commentary generation, and foul recognition under a single architecture, pretrained with supervised and video–language objectives. Empirical results show state-of-the-art performance on established benchmarks and the new SoccerReplay-test, driven by the scale and quality of SoccerReplay-1988 and the spatiotemporal modeling of MatchVision. The dataset and model collectively offer a scalable, standard paradigm to advance sports AI and fan analytics in real-world soccer contexts.

Abstract

As a globally celebrated sport, soccer has attracted widespread interest from fans all over the world. This paper aims to develop a comprehensive multi-modal framework for soccer video understanding. Specifically, we make the following contributions in this paper: (i) we introduce SoccerReplay-1988, the largest multi-modal soccer dataset to date, featuring videos and detailed annotations from 1,988 complete matches, with an automated annotation pipeline; (ii) we present an advanced soccer-specific visual encoder, MatchVision, which leverages spatiotemporal information across soccer videos and excels in various downstream tasks; (iii) we conduct extensive experiments and ablation studies on event classification, commentary generation, and multi-view foul recognition. MatchVision demonstrates state-of-the-art performance on all of them, substantially outperforming existing models, which highlights the superiority of our proposed data and model. We believe that this work will offer a standard paradigm for sports understanding research.

Figures (9)

• Figure 1: Overview. We present SoccerReplay-1988, the largest soccer dataset to date, and a powerful soccer-specific visual encoder, MatchVision, capable of excelling in various tasks such as event classification and commentary generation.
• Figure 2: Automated Data Curation Pipeline. The collected soccer video data are automatically processed for temporal alignment, event summarization, and anonymization by our curation pipeline.
• Figure 3: Overview of MatchVision. (a) The model architecture and its spatiotemporal feature extraction process; (b) Details of visual encoder pretraining, including supervised classification and video-language contrastive learning; (c) Implementation details of specific heads for various downstream tasks, including commentary generation, foul recognition, and event classification.
• Figure 4: Qualitative Results for Event Classification and Commentary Generation. Here, "w/o SR" and "w/ SR" indicate models trained without and with the SoccerReplay-1988 dataset, respectively. Incorporating SoccerReplay-1988 improves event classification accuracy. Moreover, this enriched training data enables the model to demonstrate several advantages in commentary generation: (a) more detailed descriptions, (b) greater linguistic variety, (c) higher event depiction accuracy, (d) better adherence to updated rules, and (e) improved specificity in scenario response.
• Figure 5: Comprehensive Visualizations of SoccerReplay-1988 Dataset.