Breaking the Barriers: Video Vision Transformers for Word-Level Sign Language Recognition
Alexander Brettmann, Jakob Grävinghoff, Marlene Rüschoff, Marie Westhues
TL;DR
This work tackles word-level sign language recognition by leveraging transformer-based Video Vision Transformers to capture global spatiotemporal dynamics in sign-language videos. It compares TimeSformer and VideoMAE, showing that VideoMAE delivers superior accuracy on WLASL100 with a top-1 of 75.58% (top-5 91.86%, top-10 95.74%), outperforming both TimeSformer and the CNN baseline I3D, even with a relatively small dataset and modest training. The results demonstrate the data efficiency and effectiveness of masked autoencoder–based transformers for sign language video understanding, suggesting strong potential for broader adoption and scaling. The study points to future work extending to larger datasets like WLASL2000 and moving toward sentence-level recognition, potentially reducing barriers for communication with the DHH community.
Abstract
Sign language is a fundamental means of communication for the deaf and hard-of-hearing (DHH) community, enabling nuanced expression through gestures, facial expressions, and body movements. Despite its critical role in facilitating interaction within the DHH population, significant barriers persist due to the limited fluency in sign language among the hearing population. Overcoming this communication gap through automatic sign language recognition (SLR) remains a challenge, particularly at a dynamic word-level, where temporal and spatial dependencies must be effectively recognized. While Convolutional Neural Networks (CNNs) have shown potential in SLR, they are computationally intensive and have difficulties in capturing global temporal dependencies between video sequences. To address these limitations, we propose a Video Vision Transformer (ViViT) model for word-level American Sign Language (ASL) recognition. Transformer models make use of self-attention mechanisms to effectively capture global relationships across spatial and temporal dimensions, which makes them suitable for complex gesture recognition tasks. The VideoMAE model achieves a Top-1 accuracy of 75.58% on the WLASL100 dataset, highlighting its strong performance compared to traditional CNNs with 65.89%. Our study demonstrates that transformer-based architectures have great potential to advance SLR, overcome communication barriers and promote the inclusion of DHH individuals.