Uni-Sign: Toward Unified Sign Language Understanding at Scale

TL;DR

Uni-Sign tackles the gap between pre-training and downstream SLU tasks by introducing large-scale generative pre-training and a unified fine-tuning paradigm that treats ISLR, CSLR, and SLT as a single SLT task. It introduces CSL-News, a 1,985-hour CSL translation dataset, and a PGF-based fusion module with score-aware sampling to robustly fuse pose and RGB cues. Empirically, Uni-Sign achieves state-of-the-art results across multiple SLU benchmarks, demonstrating strong cross-task transfer without task-specific fine-tuning tricks. The work advances scalable, unified SLU at scale with practical implications for Deaf communities.

Abstract

Sign language pre-training has gained increasing attention for its ability to enhance performance across various sign language understanding (SLU) tasks. However, existing methods often suffer from a gap between pre-training and fine-tuning, leading to suboptimal results. To address this, we propose Uni-Sign, a unified pre-training framework that eliminates the gap between pre-training and downstream SLU tasks through a large-scale generative pre-training strategy and a novel fine-tuning paradigm. First, we introduce CSL-News, a large-scale Chinese Sign Language (CSL) dataset containing 1,985 hours of video paired with textual annotations, which enables effective large-scale pre-training. Second, Uni-Sign unifies SLU tasks by treating downstream tasks as a single sign language translation (SLT) task during fine-tuning, ensuring seamless knowledge transfer between pre-training and fine-tuning. Furthermore, we incorporate a prior-guided fusion (PGF) module and a score-aware sampling strategy to efficiently fuse pose and RGB information, addressing keypoint inaccuracies and improving computational efficiency. Extensive experiments across multiple SLU benchmarks demonstrate that Uni-Sign achieves state-of-the-art performance across multiple downstream SLU tasks. Dataset and code are available at github.com/ZechengLi19/Uni-Sign.

Figures (14)

• Figure 1: Comparison of paradigm and performance between previous SOTA pre-training methods and ours. $\mathcal{L}_{pt}$, $\mathcal{L}_{ts}$, and $\mathcal{L}_{lm}$ represent the pretext-task loss, task-specific loss, and language modeling loss, respectively. Our method could mainly adopt the pre-training parameters and a unified fine-tuning paradigm, which narrow the gap between pre-training and fine-tuning and therefore embeds versatility capability on multiple benchmarks across different downstream tasks, including ISLR, CSLR, and SLT.
• Figure 1: Summary statistics for different SLT datasets.
• Figure 2: Training recipe of each stage.
• Figure 3: Samples of videos and text annotations in the CSL-news dataset. The signer's face is masked in here to protect their privacy.
• Figure 3: ISLR results on various benchmarks. $\dagger$ denotes methods reproduced by hu2021signbert. Blue and Green denote the best results of previous methods and ours, respectively.