MPT-PAR:Mix-Parameters Transformer for Panoramic Activity Recognition
Wenqing Gan, Yan Sun, Feiran Liu, Xiangfeng Luo
TL;DR
This paper addresses panoramic activity recognition, which requires identifying individual actions, social group activities, and global activities in crowded scenes. It introduces MPT-PAR, a mixed-parameters transformer that combines parameter-sharing and parameter-independent cross-granularity aggregations (PSICGA), a spatio-temporal relation-enhanced (STRE) module, and a scene representation learning (SRL) component to integrate temporal, spatial, and global scene context. The approach yields state-of-the-art results on the JRDB-PAR dataset, with ablations demonstrating the complementary benefits of the proposed modules and the importance of scene context and temporal modeling. The work advances practical panoramic recognition by enabling mutual reinforcement across granularities and leveraging global scene cues for robust performance in complex environments.
Abstract
The objective of the panoramic activity recognition task is to identify behaviors at various granularities within crowded and complex environments, encompassing individual actions, social group activities, and global activities. Existing methods generally use either parameter-independent modules to capture task-specific features or parameter-sharing modules to obtain common features across all tasks. However, there is often a strong interrelatedness and complementary effect between tasks of different granularities that previous methods have yet to notice. In this paper, we propose a model called MPT-PAR that considers both the unique characteristics of each task and the synergies between different tasks simultaneously, thereby maximizing the utilization of features across multi-granularity activity recognition. Furthermore, we emphasize the significance of temporal and spatial information by introducing a spatio-temporal relation-enhanced module and a scene representation learning module, which integrate the the spatio-temporal context of action and global scene into the feature map of each granularity. Our method achieved an overall F1 score of 47.5\% on the JRDB-PAR dataset, significantly outperforming all the state-of-the-art methods.