Bengali Sign Language Recognition through Hand Pose Estimation using Multi-Branch Spatial-Temporal Attention Model

TL;DR

The paper tackles Bangla Sign Language recognition using privacy-preserving hand skeleton data and introduces a lightweight, multi-branch spatial-temporal attention model built on depthwise separable temporal convolutions. The approach uses MediaPipe to extract 21 hand joints per frame, encodes temporal dynamics with Sep-TCN, and fuses three attention streams (spatial, temporal, spatial-temporal) via multi-head attention with position embeddings and masking. It introduces BAUST-BSL-38, a new large-scale dataset, and evaluates on BdSL-38 and KU-BdSL, demonstrating strong intra- and merged-dataset generalization while maintaining significantly lower parameters and FLOPs than state-of-the-art methods. The results indicate the method is well-suited for real-time, privacy-focused BSL recognition across diverse environments, with planned expansion to a larger-scale dataset to address inter-dataset generalization gaps.

Abstract

Hand gesture-based sign language recognition (SLR) is one of the most advanced applications of machine learning, and computer vision uses hand gestures. Although, in the past few years, many researchers have widely explored and studied how to address BSL problems, specific unaddressed issues remain, such as skeleton and transformer-based BSL recognition. In addition, the lack of evaluation of the BSL model in various concealed environmental conditions can prove the generalized property of the existing model by facing daily life signs. As a consequence, existing BSL recognition systems provide a limited perspective of their generalisation ability as they are tested on datasets containing few BSL alphabets that have a wide disparity in gestures and are easy to differentiate. To overcome these limitations, we propose a spatial-temporal attention-based BSL recognition model considering hand joint skeletons extracted from the sequence of images. The main aim of utilising hand skeleton-based BSL data is to ensure the privacy and low-resolution sequence of images, which need minimum computational cost and low hardware configurations. Our model captures discriminative structural displacements and short-range dependency based on unified joint features projected onto high-dimensional feature space. Specifically, the use of Separable TCN combined with a powerful multi-head spatial-temporal attention architecture generated high-performance accuracy. The extensive experiments with a proposed dataset and two benchmark BSL datasets with a wide range of evaluations, such as intra- and inter-dataset evaluation settings, demonstrated that our proposed models achieve competitive performance with extremely low computational complexity and run faster than existing models.

Figures (14)

• Figure 1: Example of our collected dataset.
• Figure 2: Sample images of the 38 BdSL Dataset.
• Figure 3: Sample images of the KU BdSL Dataset Images.
• Figure 4: Working Flow Architecture.
• Figure 5: Mediapipe key points.