Beyond a Single Reference: Training and Evaluation with Paraphrases in Sign Language Translation

TL;DR

This work tackles the limitation of single-reference SLT data by using Large Language Models to automatically generate paraphrased translations, creating synthetic multi-reference signals for both training and evaluation. It systematically compares paraphrase generation models with ParaScore-based quality and evaluates their impact on a pose-based T5 SLT model trained on YouTubeASL and How2Sign. The results show that paraphrases do not help training when naively used as data augmentation, but improve evaluation when used as references, motivating the introduction of a new metric, $BLEU_{para}$, that correlates better with human judgments. The authors release all paraphrases, generation code, and evaluation scripts to facilitate reproducible SLT research.

Abstract

Most Sign Language Translation (SLT) corpora pair each signed utterance with a single written-language reference, despite the highly non-isomorphic relationship between sign and spoken languages, where multiple translations can be equally valid. This limitation constrains both model training and evaluation, particularly for n-gram-based metrics such as BLEU. In this work, we investigate the use of Large Language Models to automatically generate paraphrased variants of written-language translations as synthetic alternative references for SLT. First, we compare multiple paraphrasing strategies and models using an adapted ParaScore metric. Second, we study the impact of paraphrases on both training and evaluation of the pose-based T5 model on the YouTubeASL and How2Sign datasets. Our results show that naively incorporating paraphrases during training does not improve translation performance and can even be detrimental. In contrast, using paraphrases during evaluation leads to higher automatic scores and better alignment with human judgments. To formalize this observation, we introduce BLEUpara, an extension of BLEU that evaluates translations against multiple paraphrased references. Human evaluation confirms that BLEUpara correlates more strongly with perceived translation quality. We release all generated paraphrases, generation and evaluation code to support reproducible and more reliable evaluation of SLT systems.

Figures (3)

• Figure 1: ParaScore distributions for paraphrases generated by different LLMs. Kernel density estimates over all generated paraphrases (same prompt and decoding for all models). Dashed vertical lines mark per-model means ($\mu$; shown in the legend). Higher scores indicate paraphrases that better preserve meaning while avoiding near-copies, enabling a direct quality comparison across paraphrasing models.
• Figure 2: Effect of adding video-level context to the paraphrasing prompt. ParaScore distributions for Llama-3.2-3B-Instruct with sentence-only prompting versus prompting augmented with short preceding context from the same video clip. Dashed lines denote means ($\mu$); providing context shifts the distribution left and increases variance, suggesting that extra discourse information can encourage looser rewrites that more often drift from the original meaning under our ParaScore criterion.
• Figure 3: Sequential vs. iterative paraphrase generation. ParaScore distributions for Llama-3.2-3B-Instruct when producing five paraphrases in a single call (sequential) versus generating them one-by-one with feedback from earlier outputs (iterative). Histograms and density curves largely overlap, and the mean/variance (dashed lines; $\mu,\sigma$ in legend) are nearly identical, indicating that iterative prompting does not materially change paraphrase quality in our setting.