VGG Induced Deep Hand Sign Language Detection

TL;DR

This work tackles hand gesture recognition for assistive HCI by adapting a VGG-16 CNN via transfer learning and targeted fine-tuning. The system uses ImageNet pretraining, data augmentation, and a MediaPipe-based pipeline to extract hand keypoints, validating on the NUS 10-class dataset and achieving 98.33% accuracy on real-time testing with 10 alphabet gestures. The approach demonstrates that combining pre-trained CNN features with augmentation and a robust hand-tracking pipeline yields high accuracy in sign-language related tasks, offering a practical route toward real-time, accessible gesture-based interfaces.

Abstract

Hand gesture recognition is an important aspect of human-computer interaction. It forms the basis of sign language for the visually impaired people. This work proposes a novel hand gesture recognizing system for the differently-abled persons. The model uses a convolutional neural network, known as VGG-16 net, for building a trained model on a widely used image dataset by employing Python and Keras libraries. Furthermore, the result is validated by the NUS dataset, consisting of 10 classes of hand gestures, fed to the model as the validation set. Afterwards, a testing dataset of 10 classes is built by employing Google's open source Application Programming Interface (API) that captures different gestures of human hand and the efficacy is then measured by carrying out experiments. The experimental results show that by combining a transfer learning mechanism together with the image data augmentation, the VGG-16 net produced around 98% accuracy.

Figures (3)

• Figure 1: Hand Gesture Classification Model
• Figure 2: Various Inputs depicting the Alphabets ('a' - 'j')
• Figure 3: Accuracy and Loss Graph of the Model