DiffSLT: Enhancing Diversity in Sign Language Translation via Diffusion Model

TL;DR

DiffSLT is proposed, a novel gloss-free SLT framework that leverages a diffusion model, enabling diverse translations while preserving sign language semantics, and DiffSLT-P, a DiffSLT variant that conditions on pseudo-glosses and visual features, reducing the modality gap.

Abstract

Sign language translation (SLT) is challenging, as it involves converting sign language videos into natural language. Previous studies have prioritized accuracy over diversity. However, diversity is crucial for handling lexical and syntactic ambiguities in machine translation, suggesting it could similarly benefit SLT. In this work, we propose DiffSLT, a novel gloss-free SLT framework that leverages a diffusion model, enabling diverse translations while preserving sign language semantics. DiffSLT transforms random noise into the target latent representation, conditioned on the visual features of input video. To enhance visual conditioning, we design Guidance Fusion Module, which fully utilizes the multi-level spatiotemporal information of the visual features. We also introduce DiffSLT-P, a DiffSLT variant that conditions on pseudo-glosses and visual features, providing key textual guidance and reducing the modality gap. As a result, DiffSLT and DiffSLT-P significantly improve diversity over previous gloss-free SLT methods and achieve state-of-the-art performance on two SLT datasets, thereby markedly improving translation quality.

Figures (12)

• Figure 1: Translation results on the PHOENIX14T Camgoz_2018_CVPR. DiffSLT generates multiple high-quality translations that are both diverse and accurate, selecting the sentence closest to the ground truth. In contrast, existing methods produce a single translation for a sign language video. Blue indicates a correct translation, purple represents an incorrect translation, and yellow denotes cases where different words with the same meaning or alternative word choices appear across translation candidates, all conveying the same underlying meaning.
• Figure 2: Comparison of diversity scores and distributions of translated spoken sentences. Previous SLT models exhibit relatively low diversity scores, with the predicted spoken sentences showing a distorted distribution in the text embedding space compared to the ground truth. In contrast, the proposed DiffSLT demonstrates a diversity score and distribution that closely resemble those of the ground truth.
• Figure 3: Overall training framework of DiffSLT. Our training process consists of two phases: pretraining for diffusion and diffusion training. During pretraining, we extract text-aligned visual features and latent representations of spoken sentences. In the diffusion training stage, our denoising network generates target sentence latent conditioned on the visual features obtained from pretraining.
• Figure 4: Illustration of Guidance Fusion Module. GFM provides unified representations by integrating two distinct levels of visual features, $F_v$ and $F_w$. In the weakly gloss-free setting (DiffSLT-P), $F_p$ replaces $F_w$ as the input to the GFM.
• Figure 5: Qualitative results for long-context sentences from the test set of PHOENIX14T. Incorrect translations are highlighted in red, while accurate translations are highlighted in green.