Pose is all you need: The pose only group activity recognition system (POGARS)

TL;DR

This work addresses the challenge of group activity recognition by proposing POGARS, a pose-only system that relies on tracked human poses and position tracklets rather than RGB data. It uses 1D CNNs to model temporal dynamics, augmented with a temporal self-attention mechanism and a spatial attention mechanism to identify key actors, alongside multitask learning for simultaneous group and individual action recognition. The approach achieves competitive accuracy on a public volleyball dataset (e.g., 93.2% for the group task) and demonstrates stronger generalization to venue changes than RGB-based models; ball tracklets provide modest improvements. The findings suggest pose-based representations can offer robust, privacy-preserving, and data-efficient group activity recognition with good transferability across domains and environments.

Abstract

We introduce a novel deep learning based group activity recognition approach called the Pose Only Group Activity Recognition System (POGARS), designed to use only tracked poses of people to predict the performed group activity. In contrast to existing approaches for group activity recognition, POGARS uses 1D CNNs to learn spatiotemporal dynamics of individuals involved in a group activity and forgo learning features from pixel data. The proposed model uses a spatial and temporal attention mechanism to infer person-wise importance and multi-task learning for simultaneously performing group and individual action classification. Experimental results confirm that POGARS achieves highly competitive results compared to state-of-the-art methods on a widely used public volleyball dataset despite only using tracked pose as input. Further our experiments show by using pose only as input, POGARS has better generalization capabilities compared to methods that use RGB as input.

Figures (7)

• Figure 1: Overview of pose only group activity recognition system (POGARS).
• Figure 2: Different person level fusion approaches of POGARS. Each approach was tested with two temporal evolution modeling variations: bidirectional LSTMs and 1D CNNs.
• Figure 3: Composition of a single convolutional block of POGARS.
• Figure 4: Adding ball as an input modality for POGARS.
• Figure 5: Average attention weights assigned for each temporal instance for the videos in volleyball dataset.