Improvement in Sign Language Translation Using Text CTC Alignment

TL;DR

The paper tackles non-monotonic alignment in sign language translation (SLT) by proposing a Joint CTC/Attention framework with a hierarchical encoder and transfer learning to bridge vision-language gaps. It integrates gloss-oriented CTC (GlsCTC) and text-oriented TxtCTC into a two-stage training regime and a joint decoding process, with a multi-task loss L_SLT = $\lambda_1 L_{GlsCTC} + \lambda_2 L_{TxtCTC} + \lambda_3 L_{MLE}$, to handle length adjustment and reordering between sign videos $\mathcal{V}$, glosses $\mathcal{G}$, and spoken text $\mathcal{T}$ via text-based CTC alignment. The architecture comprises a Sign Embedding module, a hierarchical encoder (GlsEnc and TxtEnc), and a Joint Decoding module that balances TxtCTC and attention signals during beam search, while transfer learning via warm-start and fine-tuning mitigates data scarcity. Experiments on PHOENIX14T and CSL-Daily show the method is competitive with state-of-the-art baselines and notably improves over pure-attention models, with TxtCTC enabling gloss-free SLT in some configurations. This work highlights the viability of text-based CTC alignment for SLT and outlines directions toward scalable gloss-free SLT using CTC-guided reordering and transfer learning.

Abstract

Current sign language translation (SLT) approaches often rely on gloss-based supervision with Connectionist Temporal Classification (CTC), limiting their ability to handle non-monotonic alignments between sign language video and spoken text. In this work, we propose a novel method combining joint CTC/Attention and transfer learning. The joint CTC/Attention introduces hierarchical encoding and integrates CTC with the attention mechanism during decoding, effectively managing both monotonic and non-monotonic alignments. Meanwhile, transfer learning helps bridge the modality gap between vision and language in SLT. Experimental results on two widely adopted benchmarks, RWTH-PHOENIX-Weather 2014 T and CSL-Daily, show that our method achieves results comparable to state-of-the-art and outperforms the pure-attention baseline. Additionally, this work opens a new door for future research into gloss-free SLT using text-based CTC alignment.

Figures (3)

• Figure 1: An overview of sign language recognition and sign language translation alignment. CTC is used for monotonic alignment between the sign language video $\mathcal{V}$ and a gloss sequence $\mathcal{G}$. While the alignment between the sign language video with corresponding spoken text $\mathcal{T}$ is non-monotonic.
• Figure 2: Overview of the proposed method: Stage 1: warm-start training using multiple sign embeddings with a hierarchical encoder and joint decoding. Stage 2: fine-tuning the model with a single sign embedding.
• Figure 3: The shared SLT encoders (left): sign representations are down-sampled by a shared encoder using gloss CTC; hierarchical SLT encoders (right): sign representations are first down-sampled by GlsEnc using gloss CTC (GlsCTC) and then reordered by TxtEnc with text CTC (TxtCTC).