Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SONAR-SLT: Multilingual Sign Language Translation via Language-Agnostic Sentence Embedding Supervision

Yasser Hamidullah, Shakib Yazdani, Cennet Oguz, Josef van Genabith, Cristina España-Bonet

TL;DR

This work tackles the scalability bottleneck in sign language translation by replacing gloss-based supervision with language-agnostic, multimodal embeddings derived from SONAR, enabling direct multilingual SLT without language-specific glosses. It introduces a coupled augmentation strategy that combines multilingual target-language augmentation with video perturbations to improve robustness in low-resource settings. A decoder can generate translations into multiple languages from a single semantic vector, and decoder anchoring via auto-encoding preserves alignment to the pretrained semantic space. The approach supports fusion across sign-language datasets and demonstrates semantic-quality gains (BLEURT) over text-embedding baselines, with notable robustness in multilingual and low-resource scenarios. Overall, language-agnostic supervision paired with coupled augmentation offers a scalable, semantically grounded alternative to traditional SLT training, with open-source resources to facilitate reproducibility and further research.

Abstract

Sign language translation (SLT) is typically trained with text in a single spoken language, which limits scalability and cross-language generalization. Earlier approaches have replaced gloss supervision with text-based sentence embeddings, but up to now, these remain tied to a specific language and modality. In contrast, here we employ language-agnostic, multimodal embeddings trained on text and speech from multiple languages to supervise SLT, enabling direct multilingual translation. To address data scarcity, we propose a coupled augmentation method that combines multilingual target augmentations (i.e. translations into many languages) with video-level perturbations, improving model robustness. Experiments show consistent BLEURT gains over text-only sentence embedding supervision, with larger improvements in low-resource settings. Our results demonstrate that language-agnostic embedding supervision, combined with coupled augmentation, provides a scalable and semantically robust alternative to traditional SLT training.

SONAR-SLT: Multilingual Sign Language Translation via Language-Agnostic Sentence Embedding Supervision

TL;DR

This work tackles the scalability bottleneck in sign language translation by replacing gloss-based supervision with language-agnostic, multimodal embeddings derived from SONAR, enabling direct multilingual SLT without language-specific glosses. It introduces a coupled augmentation strategy that combines multilingual target-language augmentation with video perturbations to improve robustness in low-resource settings. A decoder can generate translations into multiple languages from a single semantic vector, and decoder anchoring via auto-encoding preserves alignment to the pretrained semantic space. The approach supports fusion across sign-language datasets and demonstrates semantic-quality gains (BLEURT) over text-embedding baselines, with notable robustness in multilingual and low-resource scenarios. Overall, language-agnostic supervision paired with coupled augmentation offers a scalable, semantically grounded alternative to traditional SLT training, with open-source resources to facilitate reproducibility and further research.

Abstract

Sign language translation (SLT) is typically trained with text in a single spoken language, which limits scalability and cross-language generalization. Earlier approaches have replaced gloss supervision with text-based sentence embeddings, but up to now, these remain tied to a specific language and modality. In contrast, here we employ language-agnostic, multimodal embeddings trained on text and speech from multiple languages to supervise SLT, enabling direct multilingual translation. To address data scarcity, we propose a coupled augmentation method that combines multilingual target augmentations (i.e. translations into many languages) with video-level perturbations, improving model robustness. Experiments show consistent BLEURT gains over text-only sentence embedding supervision, with larger improvements in low-resource settings. Our results demonstrate that language-agnostic embedding supervision, combined with coupled augmentation, provides a scalable and semantically robust alternative to traditional SLT training.
Paper Structure (39 sections, 11 equations, 3 figures, 7 tables)

This paper contains 39 sections, 11 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Sign Language Representation
  4. Large Language Models in SLT
  5. Multilingual SLT Datasets and Models
  6. Methodology
  7. System Overview
  8. Visual Feature Extraction and Encoding
  9. Semantic Alignment
  10. Target-language augmentation.
  11. Multilingual Generation from the Semantic Vector
  12. Auto-Encoding (Decoder Anchoring)
  13. Optional Contrastive Alignment
  14. Joint Training Objective
  15. Cross-Lingual and Multi-Sign Dataset Fusion
  16. ...and 24 more sections

Figures (3)

  • Figure 1: Text-only vs. language-agnostic sentence embedding supervision.
  • Figure 2: Overall architecture of our SONAR-SLT model. Visual inputs are processed through spatial and spatio- temporal encoders, fused using hwang-etal-2025-spamo and encoded into a semantic vector aligned with multilingual sentence embeddings.
  • Figure 3: Detailed architecture without the contrastive term (NCE loss).