Sign Language Recognition using Parallel Bidirectional Reservoir Computing

TL;DR

This paper tackles the challenge of deploying sign language recognition on edge devices by combining MediaPipe-based hand-keypoint extraction with a lightweight parallel bidirectional reservoir computing framework (PBRC). The approach uses two bidirectional ESN reservoirs in parallel, with ridge regression as the sole trainable component, delivering efficient training times and competitive accuracy on the WLASL100 dataset. Compared to deep-learning baselines like Bi-GRU, PBRC markedly reduces training time while maintaining solid recognition performance, highlighting its suitability for real-time edge applications. The results demonstrate the practicality of reservoir computing for SLR and point toward future work in continuous sign language and reservoir optimization to further boost accuracy and applicability.

Abstract

Sign language recognition (SLR) facilitates communication between deaf and hearing communities. Deep learning based SLR models are commonly used but require extensive computational resources, making them unsuitable for deployment on edge devices. To address these limitations, we propose a lightweight SLR system that combines parallel bidirectional reservoir computing (PBRC) with MediaPipe. MediaPipe enables real-time hand tracking and precise extraction of hand joint coordinates, which serve as input features for the PBRC architecture. The proposed PBRC architecture consists of two echo state network (ESN) based bidirectional reservoir computing (BRC) modules arranged in parallel to capture temporal dependencies, thereby creating a rich feature representation for classification. We trained our PBRC-based SLR system on the Word-Level American Sign Language (WLASL) video dataset, achieving top-1, top-5, and top-10 accuracies of 60.85%, 85.86%, and 91.74%, respectively. Training time was significantly reduced to 18.67 seconds due to the intrinsic properties of reservoir computing, compared to over 55 minutes for deep learning based methods such as Bi-GRU. This approach offers a lightweight, cost-effective solution for real-time SLR on edge devices.

Figures (6)

• Figure 1: Various signers performing different signs
• Figure 2: Basic principle of operation of ESN-based RC
• Figure 3: Working principle of bidirectional reservoir computing-based architecture
• Figure 4: Schematic diagram illustrating the working of parallel bidirectional reservoir computing
• Figure 5: Feature extraction using MediaPipe