Overcoming Topology Agnosticism: Enhancing Skeleton-Based Action Recognition through Redefined Skeletal Topology Awareness

TL;DR

The investigation revealed strong variations in joint-to-joint relationships across different actions, exposing the limitations of a single adjacency matrix in capturing the variations of relational configurations emblematic of human movement, which is addressed by proposing an efficient refinement to Graph Convolutions - the BlockGC.

Abstract

Graph Convolutional Networks (GCNs) have long defined the state-of-the-art in skeleton-based action recognition, leveraging their ability to unravel the complex dynamics of human joint topology through the graph's adjacency matrix. However, an inherent flaw has come to light in these cutting-edge models: they tend to optimize the adjacency matrix jointly with the model weights. This process, while seemingly efficient, causes a gradual decay of bone connectivity data, culminating in a model indifferent to the very topology it sought to map. As a remedy, we propose a threefold strategy: (1) We forge an innovative pathway that encodes bone connectivity by harnessing the power of graph distances. This approach preserves the vital topological nuances often lost in conventional GCNs. (2) We highlight an oft-overlooked feature - the temporal mean of a skeletal sequence, which, despite its modest guise, carries highly action-specific information. (3) Our investigation revealed strong variations in joint-to-joint relationships across different actions. This finding exposes the limitations of a single adjacency matrix in capturing the variations of relational configurations emblematic of human movement, which we remedy by proposing an efficient refinement to Graph Convolutions (GC) - the BlockGC. This evolution slashes parameters by a substantial margin (above 40%), while elevating performance beyond original GCNs. Our full model, the BlockGCN, establishes new standards in skeleton-based action recognition for small model sizes. Its high accuracy, notably on the large-scale NTU RGB+D 120 dataset, stand as compelling proof of the efficacy of BlockGCN.

Figures (6)

• Figure 1: We reveal the remaining issues of previous GCNs in \ref{['fig:tease']} and propose BlockGCN as the remedy, which improves over previous methods w.r.t. both performance and efficiency, see \ref{['fig:performance']}.
• Figure 2: Illustration of existing approaches for multi-relational modeling (top) and our proposed BlockGC with Invariance Encodings (bottom). Invariance Encodings preserve the information of skeletal structure, while BlockGC enables multi-relational modeling, at the same time slashing the redundant weights for feature projection, thanks to its block diagonal projection matrix.
• Figure 3: Visualization of the classification accuracy on a single frame of NTU RGB+D 120 Dataset samples. We train a model w/o temporal module only on a single frame, i.e., the temporally averaged frame, compared to one randomly sampled frame. The former has a much higher accuracy than the latter ($54\%$ vs. $35.5\%$), and they are both higher than random guesses ($\frac{1}{120}$).
• Figure 4: Model architecture of our BlockGCN. BlockGC captures the joint co-occurrences in the spatial dimension, whereas Temporal Convolution learns the temporal correlations.
• Figure 5: The learned Topological Invariance Encodings of our BlockGCN at each layer. It can be seen that the learned weights are diverse and adapted to different levels of semantics.