Attention vs LSTM: Improving Word-level BISINDO Recognition

TL;DR

This work tackles the accessibility gap for BISINDO sign language in Indonesian public services by comparing two AI-based recognition models: LSTM and a 1D CNN + Transformer (1DCNNTrans). The authors use a two-stage training pipeline with pretraining on the Google-ISLR corpus and fine-tuning on a BISINDO-focused dataset, incorporating Efficient Channel Attention to enhance feature representation. Results show the 1DCNNTrans model achieving higher validation accuracy (96.12%) than LSTM (94.67%), while LSTM offers substantially lower inference latency on ARM hardware; both methods exceed 90% accuracy across 50 gestures, with 1DCNNTrans providing greater stability on complex or similar-keypoint sequences. The findings support AI-driven translation and dictionary tools to promote inclusivity in public services and point toward feasible on-device deployments for deaf communities.

Abstract

Indonesia ranks fourth globally in the number of deaf cases. Individuals with hearing impairments often find communication challenging, necessitating the use of sign language. However, there are limited public services that offer such inclusivity. On the other hand, advancements in artificial intelligence (AI) present promising solutions to overcome communication barriers faced by the deaf. This study aims to explore the application of AI in developing models for a simplified sign language translation app and dictionary, designed for integration into public service facilities, to facilitate communication for individuals with hearing impairments, thereby enhancing inclusivity in public services. The researchers compared the performance of LSTM and 1D CNN + Transformer (1DCNNTrans) models for sign language recognition. Through rigorous testing and validation, it was found that the LSTM model achieved an accuracy of 94.67%, while the 1DCNNTrans model achieved an accuracy of 96.12%. Model performance evaluation indicated that although the LSTM exhibited lower inference latency, it showed weaknesses in classifying classes with similar keypoints. In contrast, the 1DCNNTrans model demonstrated greater stability and higher F1 scores for classes with varying levels of complexity compared to the LSTM model. Both models showed excellent performance, exceeding 90% validation accuracy and demonstrating rapid classification of 50 sign language gestures.

Figures (5)

• Figure 1: Proposed LSTM architecture.
• Figure 2: Proposed 1D CNN + Transformer (1DCNNTrans) architecture.
• Figure 3: Efficient Channel Attention (ECA) architecture.
• Figure 4: Multi-head Self-attention architecture.
• Figure 5: Classification Report. (Left) LSTM and (Right) 1DCNNTrans