An Advanced Deep Learning Based Three-Stream Hybrid Model for Dynamic Hand Gesture Recognition

TL;DR

The paper tackles dynamic hand gesture recognition under varying real-world conditions by proposing a three-stream hybrid model that fuses RGB pixel features with skeleton pose information. It combines ImageNet- and ResNet50V2-based pixel streams with Mediapipe-based hand landmarks, all processed by stacked LSTM/GRU modules and fused for final classification. The approach achieves an average accuracy of 98.35% on a newly created 10-class dynamic gesture dataset and demonstrates competitive performance on a benchmark dataset, with several gestures reaching perfect scores. The method shows strong potential for robust, contactless human–computer interfaces in industrial and everyday contexts, supported by extensive data augmentation and rigorous evaluation. Future work includes simplifying the architecture and extending deployment to broader real-world scenarios.

Abstract

In the modern context, hand gesture recognition has emerged as a focal point. This is due to its wide range of applications, which include comprehending sign language, factories, hands-free devices, and guiding robots. Many researchers have attempted to develop more effective techniques for recognizing these hand gestures. However, there are challenges like dataset limitations, variations in hand forms, external environments, and inconsistent lighting conditions. To address these challenges, we proposed a novel three-stream hybrid model that combines RGB pixel and skeleton-based features to recognize hand gestures. In the procedure, we preprocessed the dataset, including augmentation, to make rotation, translation, and scaling independent systems. We employed a three-stream hybrid model to extract the multi-feature fusion using the power of the deep learning module. In the first stream, we extracted the initial feature using the pre-trained Imagenet module and then enhanced this feature by using a multi-layer of the GRU and LSTM modules. In the second stream, we extracted the initial feature with the pre-trained ReseNet module and enhanced it with the various combinations of the GRU and LSTM modules. In the third stream, we extracted the hand pose key points using the media pipe and then enhanced them using the stacked LSTM to produce the hierarchical feature. After that, we concatenated the three features to produce the final. Finally, we employed a classification module to produce the probabilistic map to generate predicted output. We mainly produced a powerful feature vector by taking advantage of the pixel-based deep learning feature and pos-estimation-based stacked deep learning feature, including a pre-trained model with a scratched deep learning model for unequalled gesture detection capabilities.

Figures (9)

• Figure 1: Basic working flow of the proposed hand gesture recognition system.
• Figure 2: Example of the input images.
• Figure 3: The architecture of the ImageNet.
• Figure 4: ResNet50 model architecture.
• Figure 5: Example of the landmark information of an input image.