Pixels or Positions? Benchmarking Modalities in Group Activity Recognition

TL;DR

This work benchmarks pixels versus positions for Group Activity Recognition by introducing SoccerNet-GAR, a synchronized multimodal dataset built from 64 World Cup matches with 94,285 labeled events across 10 classes. It provides an apples-to-apples evaluation protocol for video and tracking modalities, and presents a novel role-aware graph neural network for tracking-based GAR alongside strong video baselines. Across extensive ablations, the tracking approach with positional edges and temporal attention achieves $67.2\%$ balanced accuracy, outpacing the best video baseline by $9.1$ percentage points while using drastically fewer parameters ($197{,}K$ vs $86.3{,}M$) and training faster. The results highlight the superior signal in spatial-temporal relations captured by tracking data for coordinated team actions, though per-class analyses reveal complementary strengths, motivating future multimodal fusion.

Abstract

Group Activity Recognition (GAR) is well studied on the video modality for surveillance and indoor team sports (e.g., volleyball, basketball). Yet, other modalities such as agent positions and trajectories over time, i.e. tracking, remain comparatively under-explored despite being compact, agent-centric signals that explicitly encode spatial interactions. Understanding whether pixel (video) or position (tracking) modalities leads to better group activity recognition is therefore important to drive further research on the topic. However, no standardized benchmark currently exists that aligns broadcast video and tracking data for the same group activities, leading to a lack of apples-to-apples comparison between these modalities for GAR. In this work, we introduce SoccerNet-GAR, a multimodal dataset built from the $64$ matches of the football World Cup 2022. Specifically, the broadcast videos and player tracking modalities for $94{,}285$ group activities are synchronized and annotated with $10$ categories. Furthermore, we define a unified evaluation protocol to benchmark two strong unimodal approaches: (i) a competitive video-based classifiers and (ii) a tracking-based classifiers leveraging graph neural networks. In particular, our novel role-aware graph architecture for tracking-based GAR directly encodes tactical structure through positional edges and temporal attention. Our tracking model achieves $67.2\%$ balanced accuracy compared to $58.1\%$ for the best video baseline, while training $4.25 \times$ faster with $438 \times$ fewer parameters ($197K$ \vs $86.3M$). This study provides new insights into the relative strengths of pixels and positions for group activity recognition. Overall, it highlights the importance of modality choice and role-aware modeling for GAR.

Figures (5)

• Figure 1: Overview of Our Group Activity Recognition Benchmark. Broadcast video and agent tracking modalities are processed through modality-specific backbones. The resulting representations are evaluated for group activity recognition.
• Figure 2: Class Distribution Across Each Split. The dataset is heavily skewed toward PASS ($63.3$%) while GOAL is rare ($0.2$%). The y-axis shows event counts; percentages denote proportions.
• Figure 3: Positional Graph Representation at a Single Frame. Nodes represent players at their pitch coordinates, colored by tactical role (red: goalkeeper, blue: defenders, purple: midfielders, pink: forwards). Edges follow formation structure, connecting positionally adjacent roles within each team.
• Figure 4: Balanced Accuracy vs. Parameter Count. Tracking models achieve $67.2$% with $197$K parameters, outperforming video models ($58.1$%, $86.3$M params) by $9.1$ while using $438\times$ fewer parameters.
• Figure 5: Performance vs. Number of Training Matches for Video and Tracking Modalities. Tracking achieves higher accuracies across all data regimes and plateaus earlier ($\approx35$ matches), while video does not. The gap narrows from $15.8$% ($5$ matches) to $8.7$% ($45$ matches).