Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prototype Learning for Micro-gesture Classification

Guoliang Chen, Fei Wang, Kun Li, Zhiliang Wu, Hehe Fan, Yi Yang, Meng Wang, Dan Guo

TL;DR

This work tackles micro-gesture classification by leveraging a multimodal framework that fuses RGB cues with skeletal information using a channel-wise cross-modal attention mechanism. A prototypical refinement module creates class prototypes and calibrates ambiguous samples to reduce intra-class variance and inter-class confusion, yielding robust MG representations. The method, built on PoseConv3D with RGB-skeleton fusion, achieves a Top-1 accuracy of 70.254 on iMiGUE and, via ensemble, 70.25, outperforming prior MiGA results by a notable margin. This approach demonstrates the value of prototype-aware, multimodal strategies for subtle human gesture recognition with potential benefits for real-time, context-aware systems.

Abstract

In this paper, we briefly introduce the solution developed by our team, HFUT-VUT, for the track of Micro-gesture Classification in the MiGA challenge at IJCAI 2024. The task of micro-gesture classification task involves recognizing the category of a given video clip, which focuses on more fine-grained and subtle body movements compared to typical action recognition tasks. Given the inherent complexity of micro-gesture recognition, which includes large intra-class variability and minimal inter-class differences, we utilize two innovative modules, i.e., the cross-modal fusion module and prototypical refinement module, to improve the discriminative ability of MG features, thereby improving the classification accuracy. Our solution achieved significant success, ranking 1st in the track of Micro-gesture Classification. We surpassed the performance of last year's leading team by a substantial margin, improving Top-1 accuracy by 6.13%.

Prototype Learning for Micro-gesture Classification

TL;DR

This work tackles micro-gesture classification by leveraging a multimodal framework that fuses RGB cues with skeletal information using a channel-wise cross-modal attention mechanism. A prototypical refinement module creates class prototypes and calibrates ambiguous samples to reduce intra-class variance and inter-class confusion, yielding robust MG representations. The method, built on PoseConv3D with RGB-skeleton fusion, achieves a Top-1 accuracy of 70.254 on iMiGUE and, via ensemble, 70.25, outperforming prior MiGA results by a notable margin. This approach demonstrates the value of prototype-aware, multimodal strategies for subtle human gesture recognition with potential benefits for real-time, context-aware systems.

Abstract

In this paper, we briefly introduce the solution developed by our team, HFUT-VUT, for the track of Micro-gesture Classification in the MiGA challenge at IJCAI 2024. The task of micro-gesture classification task involves recognizing the category of a given video clip, which focuses on more fine-grained and subtle body movements compared to typical action recognition tasks. Given the inherent complexity of micro-gesture recognition, which includes large intra-class variability and minimal inter-class differences, we utilize two innovative modules, i.e., the cross-modal fusion module and prototypical refinement module, to improve the discriminative ability of MG features, thereby improving the classification accuracy. Our solution achieved significant success, ranking 1st in the track of Micro-gesture Classification. We surpassed the performance of last year's leading team by a substantial margin, improving Top-1 accuracy by 6.13%.
Paper Structure (10 sections, 10 equations, 1 figure, 2 tables)

This paper contains 10 sections, 10 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Network Architecture
  4. Cross-Modal Fusion Module
  5. Prototypical Refinement Module
  6. Experiments
  7. Datasets
  8. Evaluation Metrics and Implementation Details
  9. Experimental Results
  10. Conclusion

Figures (1)

  • Figure 1: Overview of the proposed method for micro-gesture classification. The network contains the RGB and Pose branches. The cross-attention fusion module is used to explore the correlations between RGB and Pose modalities. The prototypical refinement module defines prototype representations of each fine-grained micro-gesture category during training and forces the model to calibrate ambiguous samples among different micro-gesture categories.