ConvMixFormer- A Resource-efficient Convolution Mixer for Transformer-based Dynamic Hand Gesture Recognition

TL;DR

This work tackles the computational burden of transformer self-attention in dynamic hand gesture recognition by introducing ConvMixFormer, a convolution-based token mixer that replaces attention and is paired with a Gated Depthwise FFN to control information flow. The approach processes frame-level features from a ResNet-18 backbone through a 6-stage ConvMixFormer encoder, followed by temporal pooling and classification, and supports multi-modal late fusion. Empirical results on NVGesture and Briareo demonstrate competitive or state-of-the-art accuracy with roughly half the parameters and reduced FLOPs compared to vanilla transformers, highlighting strong parameter efficiency and practicality for real-world gesture recognition. The proposed method offers a scalable, resource-efficient solution for dynamic hand gesture recognition and multimodal fusion tasks.

Abstract

Transformer models have demonstrated remarkable success in many domains such as natural language processing (NLP) and computer vision. With the growing interest in transformer-based architectures, they are now utilized for gesture recognition. So, we also explore and devise a novel ConvMixFormer architecture for dynamic hand gestures. The transformers use quadratic scaling of the attention features with the sequential data, due to which these models are computationally complex and heavy. We have considered this drawback of the transformer and designed a resource-efficient model that replaces the self-attention in the transformer with the simple convolutional layer-based token mixer. The computational cost and the parameters used for the convolution-based mixer are comparatively less than the quadratic self-attention. Convolution-mixer helps the model capture the local spatial features that self-attention struggles to capture due to their sequential processing nature. Further, an efficient gate mechanism is employed instead of a conventional feed-forward network in the transformer to help the model control the flow of features within different stages of the proposed model. This design uses fewer learnable parameters which is nearly half the vanilla transformer that helps in fast and efficient training. The proposed method is evaluated on NVidia Dynamic Hand Gesture and Briareo datasets and our model has achieved state-of-the-art results on single and multimodal inputs. We have also shown the parameter efficiency of the proposed ConvMixFormer model compared to other methods. The source code is available at https://github.com/mallikagarg/ConvMixFormer.

Figures (2)

• Figure 1: The overview of the proposed pipeline for dynamic gesture recognition. The input $X$ can be any input eg. normal, color, depth, optical flow, or infrared images. Our pipeline uses 6 ConvMixFormer stages.
• Figure 2: (a). The conventional Transformer model consists of 2 main blocks- multi-head attention (MHA) and Feedforward network (MLP). (b). The proposed ConvMixFormer model replaces the attention mechanism with a convolution layer to mix the spatial tokens with lesser parameters and uses a gate mechanism to selectively filter the information to pass to the next stage of the transformer model. Here, Ⓡ denotes the reshaping of dimensions and CT Embed represents the class token embedding.