Scaling Sign Language Translation

TL;DR

This work addresses the data and scalability bottlenecks in sign language translation by performing large-scale pretraining across noisy multilingual YouTube SLT data, multilingual MT data, and MT-augmented SLT data within a unified encoder-decoder framework built on (m/By)T5 backbones. It demonstrates that careful data, model, and language scaling, along with cross-task and cross-modal transfer, enable zero-shot SLT and achieve new state-of-the-art results on multiple open-domain benchmarks. Key contributions include a versatile clip-level pretraining setup with prompt-based tasks, extensive exploration of MT integration directions, and data augmentation through synthetic SLT, which collectively yield substantial BLEURT gains and improved cross-language transfer. The findings highlight the practical potential and remaining challenges of gloss-free, open-domain SLT at scale, emphasizing the need for broader multilingual benchmarks and high-resource pretraining. The work’s significance lies in showing that scaling both data and models—especially with augmented SLT data and MT transfer—can markedly advance open-domain SLT toward real-world utility.

Abstract

Sign language translation (SLT) addresses the problem of translating information from a sign language in video to a spoken language in text. Existing studies, while showing progress, are often limited to narrow domains and/or few sign languages and struggle with open-domain tasks. In this paper, we push forward the frontier of SLT by scaling pretraining data, model size, and number of translation directions. We perform large-scale SLT pretraining on different data including 1) noisy multilingual YouTube SLT data, 2) parallel text corpora, and 3) SLT data augmented by translating video captions to other languages with off-the-shelf machine translation models. We unify different pretraining tasks with task-specific prompts under the encoder-decoder architecture, and initialize the SLT model with pretrained (m/By)T5 models across model sizes. SLT pretraining results on How2Sign and FLEURS-ASL#0 (ASL to 42 spoken languages) demonstrate the significance of data/model scaling and cross-lingual cross-modal transfer, as well as the feasibility of zero-shot SLT. We finetune the pretrained SLT models on 5 downstream open-domain SLT benchmarks covering 5 sign languages. Experiments show substantial quality improvements over the vanilla baselines, surpassing the previous state-of-the-art (SOTA) by wide margins.

Figures (8)

• Figure 1: BLEU scores on different benchmarks: our model sets new SOTA results across benchmarks and sign languages. Note we didn't show BLEURT because not all previous studies report BLEURT.
• Figure 2: Illustration of model architecture and pretraining task for SLT. We perform large-scale pretraining and adopt multi-task learning at clip level (multiple captions) to better leverage the supervised knowledge.
• Figure 3: Pretraining performance for Baseline + MT when varying MT languages. We show BLEURT$\uparrow$ results on FLEURS-ASL#0, and set $p_{mt}=0.5$. Note MT languages are added separately instead of jointly. Results are for ByT5 Base. "X$\rightarrow$En": MT data for translation into English; "X$\leftrightarrow$En": MT data for both translation directions; "Avg": average performance over languages. MT languages are arranged in descending order from left to right based on their training data quantity.
• Figure 4: Pretraining performance for Baseline + MT when changing the mixing ratio of MT data $p_{mt}$ on FLEURS-ASL#0 (En and De) test set. We show BLEURT$\uparrow$ results as we vary $p_{mt}$ from 0.3 to 0.9.
• Figure 5: Pretraining performance for Baseline + MT when varying MT languages on How2Sign test set. We show BLEURT$\uparrow$ results and set $p_{mt}=0.5$. Note only YT-ASL and bilingual MT data are used, i.e. MT languages are added separately instead of jointly. Results are for ByT5 Base. "X$\rightarrow$En": MT data for translation into English; "X$\leftrightarrow$En": MT data for both translation directions; "Avg": average performance over languages. MT languages are arranged in descending order from left to right based on the quantity of translation data available for each language.