SportsNGEN: Sustained Generation of Realistic Multi-player Sports Gameplay

TL;DR

We present SportsNGEN, a transformer-decoder based engine that learns from continuous player and ball tracking data to generate sustained, realistic multi-agent sports gameplay. By embedding players, the ball, and contextual information into object and context tokens and treating state updates as grid-based classifications, the model yields probabilistic, long-horizon simulations that can predict rally outcomes, support counterfactual analyses, and be fine-tuned to individual players. Empirical results on professional tennis data show that sampling parameters, token components, and surface-context information crucially affect realism and calibration, with the approach extensible to football through qualitative demonstrations. The work enables coaching insights, enhanced broadcast analytics, and rapid customization, while outlining limitations around out-of-distribution scenarios and computational demands, and pointing toward broader sports applicability in future work.

Abstract

We present a transformer decoder based sports simulation engine, SportsNGEN, trained on sports player and ball tracking sequences, that is capable of generating sustained gameplay and accurately mimicking the decision making of real players. By training on a large database of professional tennis tracking data, we demonstrate that simulations produced by SportsNGEN can be used to predict the outcomes of rallies, determine the best shot choices at any point, and evaluate counterfactual or what if scenarios to inform coaching decisions and elevate broadcast coverage. By combining the generated simulations with a shot classifier and logic to start and end rallies, the system is capable of simulating an entire tennis match. We evaluate SportsNGEN by comparing statistics of the simulations with those of real matches between the same players. We show that the model output sampling parameters are crucial to simulation realism and that SportsNGEN is probabilistically well-calibrated to real data. In addition, a generic version of SportsNGEN can be customized to a specific player by fine-tuning on the subset of match data that includes that player. Finally, we show qualitative results indicating the same approach works for football.

Figures (18)

• Figure 1: Frames from a football match simulated using SportsNGEN. The panels depict a passing sequence involving 3 players. The ball is in the red circle, with an arrow depicting the play that follows. Link to video: https://youtu.be/M0kkKiGVNzk
• Figure 2: Simulated tennis rally between 2 players using 3 shots of training data as input. Frames a) - c): Training data shots. Frames d) - f) Simulated rollout. Red and blue markings indicate player movement. The lines indicate shot trajectories. The current shot is opaque while earlier shots are more transparent. The purple line is the first simulated shot. Link to video: https://youtu.be/A1_vv12V5q0
• Figure 3: Left: SportsNGEN flow diagram. Right: Cartoons from a simulated tennis match corresponding to the flow chart steps.
• Figure 4: Top: Layout of an object token $O_{\tau, n}$. Bottom: Sequence of $T$ tokens for $M$=3 context tokens, $N$=2 players, and a ball.
• Figure 5: Visualization of the 2D and 3D classification grids used to predict the position of a player and the ball at the next time step.