Sigma: Semantically Informative Pre-training for Skeleton-based Sign Language Understanding

TL;DR

Sigma tackles three core challenges in skeleton-based sign language understanding: weak semantic grounding in visual features, imbalance between local and global information, and inefficient cross-modal learning. It introduces a unified pre-training framework with Sign-aware early fusion (SignEF), hierarchical alignment learning (HAL), and a sign-grounded text (SGT) encoder that jointly optimizes contrastive alignment, text matching, and language modelling, using $L_{pre\text{-}train} = L_{HAL} + L_{SGT}$. The approach yields state-of-the-art results across ISLR, CSLR, and SLT benchmarks (e.g., WLASL2000, CSL-Daily, How2Sign, OpenASL) and demonstrates strong cross-task transfer with skeletal data as a stand-alone modality. The work highlights the importance of semantically grounded, cross-modal pre-training for scalable, robust SLU and points to skeletal representations as a practical, efficient foundation for future sign-language technologies.

Abstract

Pre-training has proven effective for learning transferable features in sign language understanding (SLU) tasks. Recently, skeleton-based methods have gained increasing attention because they can robustly handle variations in subjects and backgrounds without being affected by appearance or environmental factors. Current SLU methods continue to face three key limitations: 1) weak semantic grounding, as models often capture low-level motion patterns from skeletal data but struggle to relate them to linguistic meaning; 2) imbalance between local details and global context, with models either focusing too narrowly on fine-grained cues or overlooking them for broader context; and 3) inefficient cross-modal learning, as constructing semantically aligned representations across modalities remains difficult. To address these, we propose Sigma, a unified skeleton-based SLU framework featuring: 1) a sign-aware early fusion mechanism that facilitates deep interaction between visual and textual modalities, enriching visual features with linguistic context; 2) a hierarchical alignment learning strategy that jointly maximises agreements across different levels of paired features from different modalities, effectively capturing both fine-grained details and high-level semantic relationships; and 3) a unified pre-training framework that combines contrastive learning, text matching and language modelling to promote semantic consistency and generalisation. Sigma achieves new state-of-the-art results on isolated sign language recognition, continuous sign language recognition, and gloss-free sign language translation on multiple benchmarks spanning different sign and spoken languages, demonstrating the impact of semantically informative pre-training and the effectiveness of skeletal data as a stand-alone solution for SLU.

Figures (12)

• Figure 1: Overview of Sigma. (a) SignEF enhances visual-linguistic alignment by injecting cross-modal features into sign and text encoders. (b) HAL is used to maximise global and local cluster agreement. (c) SGT encoder jointly optimises sign-text matching and language modelling. During fine-tuning, both the sign and SGT encoders are reused across SLU tasks.
• Figure 2: SignEF promotes progressive visual-linguistic interaction with parameters ${ W^{(x, L)}, W^{(x, N)} : x \in \{q, v, \text{out} \} }$, analogous to query, value, and output projections by vaswani2017attention.
• Figure 3: The overview of the cluster aggregator module. It converts sub-word token embeddings into cluster-level representations by grouping tokens, mapping them with offset indices, and aggregating hidden features for semantic alignment with visual inputs (See Appendix \ref{['sec:cu_aggregator']} for details).
• Figure 4: The architecture of the SGT encoder, which consists of two paths: theSTM path injects sign features via cross-attention for semantic alignment, and the LM path preserves linguistic fluency through standard transformer layers (See Appendix \ref{['sec:sgt_enc']} for details).
• Figure 5: Visualization derived from the WLASL2000 dataset. The right hand, along with its primary motion trajectory, is highlighted to illustrate the gesture dynamics. The figure shows two sign sequences, “Always” and “Someone.” Although both gestures exhibit similar hand shapes and motion trajectories, they differ in spatial and temporal extent. Disambiguating them requires not only local visual detail but also global temporal understanding and accurate alignment with linguistic meaning, highlighting the need for effective multimodal representation learning.