Beyond Pixels: Leveraging the Language of Soccer to Improve Spatio-Temporal Action Detection in Broadcast Videos

TL;DR

This paper reframes spatio-temporal action detection in soccer as a denoising sequence transduction problem that leverages long-range game-state context. By combining prior per-frame, per-player predictions with role-based game-state representations in a Transformer encoder–decoder, the method learns to reconstruct coherent, tactically plausible action sequences that better reflect actual gameplay. Empirical results on Footovision datasets show notable gains in both precision and recall in high-recall regimes, especially when incorporating game-state information and longer temporal context. The approach is designed to assist annotation work and could be enhanced further by integrating vision foundation features for richer semantic cues.

Abstract

State-of-the-art spatio-temporal action detection (STAD) methods show promising results for extracting soccer events from broadcast videos. However, when operated in the high-recall, low-precision regime required for exhaustive event coverage in soccer analytics, their lack of contextual understanding becomes apparent: many false positives could be resolved by considering a broader sequence of actions and game-state information. In this work, we address this limitation by reasoning at the game level and improving STAD through the addition of a denoising sequence transduction task. Sequences of noisy, context-free player-centric predictions are processed alongside clean game state information using a Transformer-based encoder-decoder model. By modeling extended temporal context and reasoning jointly over team-level dynamics, our method leverages the "language of soccer" - its tactical regularities and inter-player dependencies - to generate "denoised" sequences of actions. This approach improves both precision and recall in low-confidence regimes, enabling more reliable event extraction from broadcast video and complementing existing pixel-based methods.

Figures (1)

• Figure 1: Overview of our method. A STAD model first produces per-frame, per-player action predictions over a long sequence. These predictions are concatenated in a consistent role-based order, using metadata. Each token is constructed by combining these structured prior predictions with additional game-state features such as player positions and velocities. The resulting sequence of tokens is fed into an encoder-decoder Transformer, which is trained to auto-regressively generate cleaned action sequences—predicting, for each action, its class, associated player, and frame number. Our method emphasizes the importance of both temporal and inter-player context, leveraging a wide temporal window and structured game-state information.