TranSPORTmer: A Holistic Approach to Trajectory Understanding in Multi-Agent Sports

TL;DR

Evaluations show that TranSPORTmer outperforms state-of-the-art task-specific models in player forecasting, player forecasting-imputation, ball inference, and ball imputation, and its application to the intricate dynamics of multi-agent sports scenarios like soccer and basketball is showcased.

Abstract

Understanding trajectories in multi-agent scenarios requires addressing various tasks, including predicting future movements, imputing missing observations, inferring the status of unseen agents, and classifying different global states. Traditional data-driven approaches often handle these tasks separately with specialized models. We introduce TranSPORTmer, a unified transformer-based framework capable of addressing all these tasks, showcasing its application to the intricate dynamics of multi-agent sports scenarios like soccer and basketball. Using Set Attention Blocks, TranSPORTmer effectively captures temporal dynamics and social interactions in an equivariant manner. The model's tasks are guided by an input mask that conceals missing or yet-to-be-predicted observations. Additionally, we introduce a CLS extra agent to classify states along soccer trajectories, including passes, possessions, uncontrolled states, and out-of-play intervals, contributing to an enhancement in modeling trajectories. Evaluations on soccer and basketball datasets show that TranSPORTmer outperforms state-of-the-art task-specific models in player forecasting, player forecasting-imputation, ball inference, and ball imputation. https://youtu.be/8VtSRm8oGoE

Figures (5)

• Figure 1: TranSPORTmer is a holistic model that is able to perform multiple tasks for trajectory understanding in multi-agent sport scenarios. The images showcase examples using soccer and basketball data for the tasks of forecasting: predicting future trajectories given past observations; imputation: predicting agent trajectories given partial observations; inference: predicting the trajectory of an unobserved agent given the state of other ones; and state classification: assigning a semantic label to each frame of the sequence. Continuous and dashed lines correspond to observed states and predicted trajectories, respectively.
• Figure 2: TranSPORTmer. The architecture uses sequential Set Attention Blocks for attention in both temporal (SAB$_T$) and social (SAB$_S$) axes. A Positional Encoder (PE) precedes each encoder to maintain the temporal sequence. The mask $\mathbf{M}$ identifies the values to be predicted (dashed arrow), forming the complete observation tensor $\mathbf{X}_{1:T}$. The extended mask $\bar{\mathbf{M}}$ is applied to the 2 $\times$ SAB$_T$ of the first Encoder$_c$, conveying information about hidden and visible states. Blue-gray segments are involved in state classification, including the CLS extra agent and the final classification head to rank the state classes per frame. (c) operation stands for concatenation and (s) for split.
• Figure 3: Binary mask ($\mathbf{M}$) and learnable uncertainty mask ($\mathbf{M_{unc}}$) for a single agent. Null values indicate visible observations.
• Figure 4: Qualitative evaluation in soccer player forecasting and ball inference. Top: Offensive player trajectory forecasting with a time horizon of 6.4s using a prior of 3.2s. Bottom: Ball inference through the full 9.6s sequence.
• Figure 5: Left: Confusion matrix in state classification. Offensive player trajectory forecasting (left) and ball inference (right). Right: Attention maps for the ball. Visualization of attention maps in each social SAB$_S$ across agents and time for the sequences #1 and #2 in Fig. \ref{['fig:FigQual1']}-bottom (animations in suppl video).