Data-Free Class-Incremental Gesture Recognition with Prototype-Guided Pseudo Feature Replay

TL;DR

This paper tackles data-free class-incremental gesture recognition, addressing privacy and memory constraints while mitigating catastrophic forgetting. It introduces the Prototype-Guided Pseudo Feature Replay (PGPFR) framework, comprising PFGBP for online pseudo-feature generation with batch prototypes, Variational Prototype Replay (VPR) to align old-class prototypes with classifier weights using covariances, Truncated Cross-Entropy (TCE) to handle domain differences for new classes, and Continual Classifier Re-Training (CCRT) to stabilize features by keeping the backbone fixed. The approach achieves state-of-the-art results on SHREC 2017 3D and EgoGesture 3D, with substantial gains in mean Global Accuracy and reductions in IFM compared to data-free baselines, while maintaining data privacy and low spatial complexity. These findings advance practical data-free continual learning for gesture recognition and highlight the importance of prototype-aware replay and dedicated new-class optimization for robust, scalable open-set recognition in 3D gesture domains.

Abstract

Gesture recognition is an important research area in the field of computer vision. Most gesture recognition efforts focus on close-set scenarios, thereby limiting the capacity to effectively handle unseen or novel gestures. We aim to address class-incremental gesture recognition, which entails the ability to accommodate new and previously unseen gestures over time. Specifically, we introduce a Prototype-Guided Pseudo Feature Replay (PGPFR) framework for data-free class-incremental gesture recognition. This framework comprises four components: Pseudo Feature Generation with Batch Prototypes (PFGBP), Variational Prototype Replay (VPR) for old classes, Truncated Cross-Entropy (TCE) for new classes, and Continual Classifier Re-Training (CCRT). To tackle the issue of catastrophic forgetting, the PFGBP dynamically generates a diversity of pseudo features in an online manner, leveraging class prototypes of old classes along with batch class prototypes of new classes. Furthermore, the VPR enforces consistency between the classifier's weights and the prototypes of old classes, leveraging class prototypes and covariance matrices to enhance robustness and generalization capabilities. The TCE mitigates the impact of domain differences of the classifier caused by pseudo features. Finally, the CCRT training strategy is designed to prevent overfitting to new classes and ensure the stability of features extracted from old classes. Extensive experiments conducted on two widely used gesture recognition datasets, namely SHREC 2017 3D and EgoGesture 3D, demonstrate that our approach outperforms existing state-of-the-art methods by 11.8\% and 12.8\% in terms of mean global accuracy, respectively. The code is available on https://github.com/sunao-101/PGPFR-3/.

Figures (5)

• Figure 1: Illustration of the proposed data-free class-incremental gesture recognition. The main characteristics of this approach are generating pseudo features in an online fashion during the batch training of new class samples, leveraging prototypes and covariance matrices from old classes.
• Figure 2: Framework of the proposed data-free class-incremental gesture recognition with Prototype-Guided Pseudo Feature Replay (PGPFR). Backbone is fixed after being trained on $\mathcal{T}_0$. In each subsequent class increment task $\mathcal{T}_i$, we use class prototypes and covariance matrices to generate pseudo features of old classes through PFGBP, only adjusting classification heads. For the acquisition of new class knowledge, we minimize $\mathcal{L}_T$ to learn the decision boundaries of the new class; For preserving knowledge of old classes, we minimize $\mathcal{L}_P$ and $\mathcal{L}_V$ to strengthen the decision boundaries of old classes.
• Figure 3: Comparison of accuracy of new and old classes among ablated approaches for each task on the SHREC 2017 3D dataset.
• Figure 4: T-SNE visualization of the deep features extracted from the SHREC 2017 3D dataset. The experiemnts of class-incremental learning are conducted with 7 tasks. (a) depicts the features of the first 8 classes from PGPFR after training on task 0, (b) shows the features of the same first 8 classes from PGPFR but after training on task 6, (c) illustrates the features of all 14 classes from PGPFR after training on task 6. In all visualizations, data points are colored according to their corresponding ground-truth labels.
• Figure 5: Visualization of gesture samples and corresponding classification results between the proposed method and BOAT-MI aich2023data. For a given sample, the gesture skeleton sequence is represented on the left, while the confidence scores predicted by different methods are displayed on the right.