WLASL-LEX: a Dataset for Recognising Phonological Properties in American Sign Language

TL;DR

This work addresses the gap in phonology-aware processing for Sign Language Processing by introducing Phonological Property Recognition (PPR) and constructing WLASL-Lex2001, a large-scale dataset of ASL videos annotated with six manual phonological properties drawn from ASL-Lex. By cross-referencing ASL-Lex with the WLASL dataset and extracting skeletal features via FrankMocap and HRNet, the authors train multiple models, finding that a Spatio-Temporal Graph Convolutional Network (STGCN) with HRNet features achieves the strongest results and generalizes to signs unseen during training. The study demonstrates the feasibility of data-driven phonology recognition in sign languages, highlights the benefits of structured skeleton inputs, and identifies avenues for improvement such as joint property learning and phonology-guided tokenization for continuous sign language processing. The dataset and findings provide a resource and baseline for phonology-aware SLP research, with potential implications for rapid annotation and cross-language phonological studies.

Abstract

Signed Language Processing (SLP) concerns the automated processing of signed languages, the main means of communication of Deaf and hearing impaired individuals. SLP features many different tasks, ranging from sign recognition to translation and production of signed speech, but has been overlooked by the NLP community thus far. In this paper, we bring to attention the task of modelling the phonology of sign languages. We leverage existing resources to construct a large-scale dataset of American Sign Language signs annotated with six different phonological properties. We then conduct an extensive empirical study to investigate whether data-driven end-to-end and feature-based approaches can be optimised to automatically recognise these properties. We find that, despite the inherent challenges of the task, graph-based neural networks that operate over skeleton features extracted from raw videos are able to succeed at the task to a varying degree. Most importantly, we show that this performance pertains even on signs unobserved during training.

Figures (1)

• Figure 1: We annotate ASL sign videos with their corresponding phonological information and skeleton features of the speakers, and train neural networks to recognise the former from the latter.