Patch as Node: Human-Centric Graph Representation Learning for Multimodal Action Recognition

TL;DR

PAN introduces a human-centric graph representation for multimodal action recognition by sampling RGB patch embeddings around 2D skeleton joints to form spatiotemporal graphs. It adds attention-based post calibration to reduce reliance on high-quality skeletal data and proposes two fusion variants, PAN-Ensemble and PAN-Unified, achieving state-of-the-art results on multiple benchmarks. The approach aligns RGB-based token graphs with skeletal graphs, enabling fine-grained cross-modal fusion while maintaining temporal alignment with skeleton-based methods. Extensive experiments and ablations validate the effectiveness of guided sampling, graph-based processing, and the proposed fusion schemes, highlighting potential for online action recognition.

Abstract

While human action recognition has witnessed notable achievements, multimodal methods fusing RGB and skeleton modalities still suffer from their inherent heterogeneity and fail to fully exploit the complementary potential between them. In this paper, we propose PAN, the first human-centric graph representation learning framework for multimodal action recognition, in which token embeddings of RGB patches containing human joints are represented as spatiotemporal graphs. The human-centric graph modeling paradigm suppresses the redundancy in RGB frames and aligns well with skeleton-based methods, thus enabling a more effective and semantically coherent fusion of multimodal features. Since the sampling of token embeddings heavily relies on 2D skeletal data, we further propose attention-based post calibration to reduce the dependency on high-quality skeletal data at a minimal cost interms of model performance. To explore the potential of PAN in integrating with skeleton-based methods, we present two variants: PAN-Ensemble, which employs dual-path graph convolution networks followed by late fusion, and PAN-Unified, which performs unified graph representation learning within a single network. On three widely used multimodal action recognition datasets, both PAN-Ensemble and PAN-Unified achieve state-of-the-art (SOTA) performance in their respective settings of multimodal fusion: separate and unified modeling, respectively.

Figures (5)

• Figure 1: PAN constructs visual token graphs based on 2D skeletal data. The learned representations can effectively distinguish five actions which are challenging for skeletal graphs, even when viewpoint changes lead to visually similar appearances.
• Figure 2: Primary directions of integrating RGB and skeleton modalities.
• Figure 3: Overview of PAN. 2D skeletal data and the index mapping are only required for guided sampling, whereas uniform sampling relies solely on RGB frames. In the attention-based post calibration, the sampled and original token embeddings serve as query and key / value respectively. $L_1$ basic blocks of GCN are then stacked, followed by a classification head. Here GC and TC denote graph convolution and temporal convolution, respectively.
• Figure 4: Left: PAN-Unified, where visual token graph embeddings and skeletal graph embeddings are modeled in a single GCN. Right: PAN-Ensemble, in which late fusion is applied by summing the classification scores.
• Figure 5: Visualization of attention maps on three actions. Left: Guided Sampling; Right: Even Sampling. The actions are punching other person, drinking water and wearing a shoe.