Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GestFormer: Multiscale Wavelet Pooling Transformer Network for Dynamic Hand Gesture Recognition

Mallika Garg, Debashis Ghosh, Pyari Mohan Pradhan

TL;DR

GestFormer tackles dynamic hand gesture recognition by introducing a resource-efficient Multiscale Wavelet Pooling Transformer (MWPT) that replaces standard attention with pooling-based token mixing. The model leverages Multiscale Wavelet Pooling Attention (MWPA) to operate in the wavelet domain, followed by a Gated Depthwise FFN (GDFN) to gate detailed features, and employs multiscale pooling to capture scale variations. Integrated with ResNet-18 feature extraction and a simple late fusion strategy for multimodal inputs, GestFormer achieves state-of-the-art performance on NVGesture and Briareo while using markedly fewer parameters and MACs than traditional transformers. The approach demonstrates that wavelet-based, pooling-centric transformers can deliver competitive accuracy for dynamic gesture recognition with improved efficiency, making it practical for real-time or resource-constrained scenarios. Code is publicly available at the provided GitHub repository.

Abstract

Transformer model have achieved state-of-the-art results in many applications like NLP, classification, etc. But their exploration in gesture recognition task is still limited. So, we propose a novel GestFormer architecture for dynamic hand gesture recognition. The motivation behind this design is to propose a resource efficient transformer model, since transformers are computationally expensive and very complex. So, we propose to use a pooling based token mixer named PoolFormer, since it uses only pooling layer which is a non-parametric layer instead of quadratic attention. The proposed model also leverages the space-invariant features of the wavelet transform and also the multiscale features are selected using multi-scale pooling. Further, a gated mechanism helps to focus on fine details of the gesture with the contextual information. This enhances the performance of the proposed model compared to the traditional transformer with fewer parameters, when evaluated on dynamic hand gesture datasets, NVidia Dynamic Hand Gesture and Briareo datasets. To prove the efficacy of the proposed model, we have experimented on single as well multimodal inputs such as infrared, normals, depth, optical flow and color images. We have also compared the proposed GestFormer in terms of resource efficiency and number of operations. The source code is available at https://github.com/mallikagarg/GestFormer.

GestFormer: Multiscale Wavelet Pooling Transformer Network for Dynamic Hand Gesture Recognition

TL;DR

GestFormer tackles dynamic hand gesture recognition by introducing a resource-efficient Multiscale Wavelet Pooling Transformer (MWPT) that replaces standard attention with pooling-based token mixing. The model leverages Multiscale Wavelet Pooling Attention (MWPA) to operate in the wavelet domain, followed by a Gated Depthwise FFN (GDFN) to gate detailed features, and employs multiscale pooling to capture scale variations. Integrated with ResNet-18 feature extraction and a simple late fusion strategy for multimodal inputs, GestFormer achieves state-of-the-art performance on NVGesture and Briareo while using markedly fewer parameters and MACs than traditional transformers. The approach demonstrates that wavelet-based, pooling-centric transformers can deliver competitive accuracy for dynamic gesture recognition with improved efficiency, making it practical for real-time or resource-constrained scenarios. Code is publicly available at the provided GitHub repository.

Abstract

Transformer model have achieved state-of-the-art results in many applications like NLP, classification, etc. But their exploration in gesture recognition task is still limited. So, we propose a novel GestFormer architecture for dynamic hand gesture recognition. The motivation behind this design is to propose a resource efficient transformer model, since transformers are computationally expensive and very complex. So, we propose to use a pooling based token mixer named PoolFormer, since it uses only pooling layer which is a non-parametric layer instead of quadratic attention. The proposed model also leverages the space-invariant features of the wavelet transform and also the multiscale features are selected using multi-scale pooling. Further, a gated mechanism helps to focus on fine details of the gesture with the contextual information. This enhances the performance of the proposed model compared to the traditional transformer with fewer parameters, when evaluated on dynamic hand gesture datasets, NVidia Dynamic Hand Gesture and Briareo datasets. To prove the efficacy of the proposed model, we have experimented on single as well multimodal inputs such as infrared, normals, depth, optical flow and color images. We have also compared the proposed GestFormer in terms of resource efficiency and number of operations. The source code is available at https://github.com/mallikagarg/GestFormer.
Paper Structure (17 sections, 4 equations, 4 figures, 6 tables)

This paper contains 17 sections, 4 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Transformer for Vision Tasks
  4. Token Mixing
  5. Transformer for Gesture Recognition
  6. Method
  7. Overview
  8. Multiscale Wavelet Pooling Transformer (MWPT)
  9. Multiscale Wavelet Pooling Attention (MWPA)
  10. Gated Depthwise Feed Forward Network (GDFN)
  11. Multi-Modal Late Fusion
  12. Experiments and Discussion
  13. Datasets
  14. Implementation Details
  15. Results and Discussion
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Some samples of depth (first row) and surface normals (last row) from the NVGesture and Briareo dataset.These samples are taken from d2020transformer.
  • Figure 2: The overall architecture of the proposed GestFormer for dynamic hand gesture recognition. The proposed GestFormer consists of Multiscale Wavelet Pyramid Attention module which comprises of Wavelet Coefficient Processing (WCP) and Multi-scale Pooling architecture (MSP) to perform learning in the wavelet coefficient space with multiscaled pooling to capture the scaled attentive information. GestFormer also leverage the Gated Dconv FFN (GDFN) to control the forward flow of the information.
  • Figure 3: Detailed Wavelet transform Processing (WCP) block. The proposed WCP requires only linear time complexity. It first maps the input into its wavelet coefficients which decomposes the input into multipple sinusoidal waves. The wavelet coefficients is the magnitude of the sinusoidal. After enhancing these magnitudes using Dconv ($DC_x$, x= 1,2,3,4), the coefficients are re-mapped in input space via backward wavelet transform.
  • Figure 4: The detailed Gated Depth-wise Feed Forward Network (GDFN) structure. GDFN facilitates subsequent layers within the network hierarchy to concentrate on more detailed image attributes, thereby resulting in better performance of the complete model.