Are Spatial-Temporal Graph Convolution Networks for Human Action Recognition Over-Parameterized?

TL;DR

The paper demonstrates that spatial-temporal graph convolution networks (ST-GCNs) for skeleton-based HAR are highly over-parameterized. It introduces a sparse-ST-GCNs generator and validates them through the lottery-ticket hypothesis, showing that sparse sub-networks can match dense performance even with substantial parameter pruning. Building on this, the authors propose multi-level sparsity by assembling backbones at different sparsity levels, achieving comparable or better HAR accuracy with roughly two-thirds fewer parameters across multiple datasets. This approach offers a practical path to deploy HAR models on resource-constrained devices and highlights the potential of sparse ensembles to enhance performance without increasing model size.

Abstract

Spatial-temporal graph convolutional networks (ST-GCNs) showcase impressive performance in skeleton-based human action recognition (HAR). However, despite the development of numerous models, their recognition performance does not differ significantly after aligning the input settings. With this observation, we hypothesize that ST-GCNs are over-parameterized for HAR, a conjecture subsequently confirmed through experiments employing the lottery ticket hypothesis. Additionally, a novel sparse ST-GCNs generator is proposed, which trains a sparse architecture from a randomly initialized dense network while maintaining comparable performance levels to the dense components. Moreover, we generate multi-level sparsity ST-GCNs by integrating sparse structures at various sparsity levels and demonstrate that the assembled model yields a significant enhancement in HAR performance. Thorough experiments on four datasets, including NTU-RGB+D 60(120), Kinetics-400, and FineGYM, demonstrate that the proposed sparse ST-GCNs can achieve comparable performance to their dense components. Even with 95% fewer parameters, the sparse ST-GCNs exhibit a degradation of <1% in top-1 accuracy. Meanwhile, the multi-level sparsity ST-GCNs, which require only 66% of the parameters of the dense ST-GCNs, demonstrate an improvement of >1% in top-1 accuracy. The code is available at https://github.com/davelailai/Sparse-ST-GCN.

Figures (6)

• Figure 1: Model comparisons across the NTURGB+D benchmarks. 'Dense' means dense backbone, the deviations of Top-1 Acc across all the four NTURGB+D benchmarks remain below 2$\%$, and the results are independent of the model size. '$80\%$ Sparse' means $80\%$ parameters are masked out, and the sparsity model shows a slight or no degradation in Top-1 Acc when compared with the corresponding dense model. 'Assemble' means multi-level sparsity ST-GCNs by incorporating the sparse structure at different sparsity levels, the results show a significant improvement compared with the corresponding dense model.
• Figure 2: The framework of sparse ST-GCNs generator. (a) represents the Lottery Ticket Perspective of Sparse ST-GCNs, where the weights $W$ in ST-GCNs are fixed, and a learnable mask $m$ is learned. The $S(m)$ represents the binary operation. (b) and (c) represent the two stages of sparse ST-GCNs generator, where the weights $W$ in ST-GCNs are learnable and the mask is pre-defined and kept fixed. During the Warm-up Stage in (b), all the parameters $W$ are involved in training, and the masked parameters $W^*$ with $M=0$ are constrained by penalty loss. During the Fine-Tuning Stage in (c), only the parameters with $M=1$ are engaged in training.
• Figure 3: Results for LTH in sparse ST-GCNs. Sparse level means the percentage of masked parameters. The sparse ST-GCNs extracted from pre-trained dense networks can achieve results comparable to the fully trained dense network. which indicates the over-parametrization of the ST-GCNs. However, a notable degradation in the top-1 accuracy is observed at high sparse levels, and the sparse ST-GCN fails to compare favourably with the baseline in the case of the randomly initiated.
• Figure 4: Ablation study of the penalty loss. 'Yes' means training the warm-up stage with the information compression penalty, and 'No' means without. It is obvious that the penalty loss had a positive influence on the performance of sparse ST-GCNs.
• Figure 5: Ablation experiment of the warm-up stage. 'Yes' means training the sparse ST-GCNs with the warm-up stage, and 'No' means training the sparse ST-GCNs without the warm-up stage, Four backbones were utilized on the NTU RGB+D dataset.