Multi-Modal Soccer Scene Analysis with Masked Pre-Training

TL;DR

The paper tackles the challenge of understanding soccer scenes by jointly inferring ball trajectory, ball state, and ball possessor from multi-modal inputs without direct ball data. It introduces a unified transformer-based architecture that fuses player trajectories, player types, and player-centric image crops, along with CropDrop, a masking pre-training strategy for visual data. Across a large-scale professional-matches dataset, the approach achieves state-of-the-art performance on all tasks, demonstrating the value of cross-modal fusion and realistic masking. This work enhances robustness in real-world broadcast settings and highlights effective strategies for multi-modal social-temporal reasoning in sports analytics.

Abstract

In this work we propose a multi-modal architecture for analyzing soccer scenes from tactical camera footage, with a focus on three core tasks: ball trajectory inference, ball state classification, and ball possessor identification. To this end, our solution integrates three distinct input modalities (player trajectories, player types and image crops of individual players) into a unified framework that processes spatial and temporal dynamics using a cascade of sociotemporal transformer blocks. Unlike prior methods, which rely heavily on accurate ball tracking or handcrafted heuristics, our approach infers the ball trajectory without direct access to its past or future positions, and robustly identifies the ball state and ball possessor under noisy or occluded conditions from real top league matches. We also introduce CropDrop, a modality-specific masking pre-training strategy that prevents over-reliance on image features and encourages the model to rely on cross-modal patterns during pre-training. We show the effectiveness of our approach on a large-scale dataset providing substantial improvements over state-of-the-art baselines in all tasks. Our results highlight the benefits of combining structured and visual cues in a transformer-based architecture, and the importance of realistic masking strategies in multi-modal learning.

Figures (2)

• Figure 1: Overview of our model to jointly infer ball possessor, ball state and ball trajectory from player trajectories, player type and image crops. After the three input modalities are projected to a common embedding, concatenated (c) and fused, two CLS tokens are appended to aggregate global information specific for their supervised tasks. The coarse and fine encoders are formed by Set Attention Blocks that act both socially and temporally. Following the splitting (s) of the tokens, different MLPs guide them to become the proper outcomes to the three tasks. A switch represents that the agent masking strategy CropDrop is only used during pre-training.
• Figure 2: Qualitative evaluation. The figure shows 4 sequences where both offensive and defensive players are displayed with the ball prediction and the corresponding ground truth motion. Markers represent the ending point of trajectories. Sequence a) depicts our model superiority in ball inference. As some passes are incorrectly hallucinated by competing approaches, ball trajectory is wrong for them, as evidenced in b). In c) competing approaches miss the one-two pass at the start of the sequence while our model correctly captures it. d) manifests that when a sequence begins or ends with the ball not in possession, our model struggles due to limited temporal context.