Skeleton-based Group Activity Recognition via Spatial-Temporal Panoramic Graph

TL;DR

The paper introduces a panoramic multi-person-object graph and MP-GCN to tackle skeleton-based group activity recognition without relying on ground-truth person boxes. By unifying intra-person, inter-person, and person-object interactions within a single graph and employing an early-fusion, multi-branch MP-GCN, the approach achieves state-of-the-art results on Volleyball and NBA while remaining computationally efficient with only 2D keypoints. It also extends to in-the-wild settings like Kinetics400, illustrating generalization and the benefits of incorporating object keypoints. The key contributions include a tracking-based pose reassignment pipeline, a panoramic graph that scales to multiple participants, and an effective four-branch architecture that fuses diverse skeletal cues for improved GAR performance. Overall, the method demonstrates that pose+object information can surpass RGB-based approaches in certain group activity tasks while reducing data and computation requirements.

Abstract

Group Activity Recognition aims to understand collective activities from videos. Existing solutions primarily rely on the RGB modality, which encounters challenges such as background variations, occlusions, motion blurs, and significant computational overhead. Meanwhile, current keypoint-based methods offer a lightweight and informative representation of human motions but necessitate accurate individual annotations and specialized interaction reasoning modules. To address these limitations, we design a panoramic graph that incorporates multi-person skeletons and objects to encapsulate group activity, offering an effective alternative to RGB video. This panoramic graph enables Graph Convolutional Network (GCN) to unify intra-person, inter-person, and person-object interactive modeling through spatial-temporal graph convolutions. In practice, we develop a novel pipeline that extracts skeleton coordinates using pose estimation and tracking algorithms and employ Multi-person Panoramic GCN (MP-GCN) to predict group activities. Extensive experiments on Volleyball and NBA datasets demonstrate that the MP-GCN achieves state-of-the-art performance in both accuracy and efficiency. Notably, our method outperforms RGB-based approaches by using only estimated 2D keypoints as input. Code is available at https://github.com/mgiant/MP-GCN

Figures (4)

• Figure 1: A group activity is represented as a panoramic graph which consists of multi-person skeletons and object keypoints.
• Figure 2: Architecture of MP-GCN and components of the basic block, where $C,C',T,T',N'$ and $K$ denote the numbers of input channels, output channels, input frames, output frames, joints, and subsets in SGC, respectively. $\odot$, $\otimes$, and © represent the matrix production, element-wise production, and concatenation, respectively.
• Figure 3: Visualization of the extracted skeletons, object keypoints and activation maps.
• Figure 4: $t$-SNE feature embedding visualization on NBA dataset for different graphs.