Beyond Words: AuralLLM and SignMST-C for Sign Language Production and Bidirectional Accessibility

TL;DR

The paper addresses bidirectional Chinese Sign Language accessibility by introducing CNText2Sign and CNSign, unified datasets linking CSL vocabulary to standardized pose data and realistic CSL contexts. It proposes AuraLLM, a decoupled SLP system that maps natural language to pose-enriched representations and renders pose-conditioned sign videos, enabling direct gesture-accuracy evaluation, and SignMST-C, a SLT model with self-supervised pretraining, multimodal fusion, and a text correction network that achieves state-of-the-art results on PHOENIX2014-T ($BLEU$-4 up to 32.08). The CNText2Sign/CNSign foundation supports all-scenario SLP and robust SLT, with direct pose evaluation via CNText2Sign and competitive metrics (e.g., $BLEU$, $ROUGE$, $CER$) demonstrating improved fidelity and linguistic coverage. Together, these contributions advance unified, bidirectional CSL accessibility and practical deployment for Deaf communities and hearing users alike.

Abstract

Sign language is the primary communication mode for 72 million hearing-impaired individuals worldwide, necessitating effective bidirectional Sign Language Production and Sign Language Translation systems. However, functional bidirectional systems require a unified linguistic environment, hindered by the lack of suitable unified datasets, particularly those providing the necessary pose information for accurate Sign Language Production (SLP) evaluation. Concurrently, current SLP evaluation methods like back-translation ignore pose accuracy, and high-quality coordinated generation remains challenging. To create this crucial environment and overcome these challenges, we introduce CNText2Sign and CNSign, which together constitute the first unified dataset aimed at supporting bidirectional accessibility systems for Chinese sign language; CNText2Sign provides 15,000 natural language-to-sign mappings and standardized skeletal keypoints for 8,643 vocabulary items supporting pose assessment. Building upon this foundation, we propose the AuraLLM model, which leverages a decoupled architecture with CNText2Sign's pose data for novel direct gesture accuracy assessment. The model employs retrieval augmentation and Cascading Vocabulary Resolution to handle semantic mapping and out-of-vocabulary words and achieves all-scenario production with controllable coordination of gestures and facial expressions via pose-conditioned video synthesis. Concurrently, our Sign Language Translation model SignMST-C employs targeted self-supervised pretraining for dynamic feature capture, achieving new SOTA results on PHOENIX2014-T with BLEU-4 scores up to 32.08. AuraLLM establishes a strong performance baseline on CNText2Sign with a BLEU-4 score of 50.41 under direct evaluation.

Figures (4)

• Figure 1: BeyondWords: Enabling Barrier-Free Communication Between Hearing and Hearing-Impaired Individuals.
• Figure 2: AuraLLM architecture overview. The Semantic-to-Representation Translation stage (upper section) converts natural language input to skeletal poses using an LLM and multi-level matching. These poses are then rendered into sign language video by the Hierarchical Sign Video Synthesis Module (lower section).
• Figure 3: An overview of SignMST-C, starts with video frames and landmarks processed through 3D ResNet18 and 1D convolution for spatial-temporal and geometric features.
• Figure 4: Demonstrating the enhanced detail and realism from Gen-3 Alpha refinement (top) over baseline ControlNet generation (bottom) for sign language video.