Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sign Language Translation with Sentence Embedding Supervision

Yasser Hamidullah, Josef van Genabith, Cristina España-Bonet

TL;DR

This work tackles the reliance on scarce gloss annotations in sign language translation by introducing sentence embedding supervision (SEM). It decomposes SLT into sign2sem and sem2text modules, enabling gloss-free and multilingual translation through pretraining on raw text and end-to-end fine-tuning, with two end-to-end variants: sign2sem2text and sign2(sem+text). Empirical results on PHOENIX-2014T (DGS) and How2Sign (ASL) show that SEM-based approaches, especially the multitask variant with a pretrained text decoder (sign2(sem+text) with mBART), achieve state-of-the-art gloss-free performance on How2Sign and surpass prior gloss-free baselines on PHOENIX-2014T, though they still lag behind gloss-based systems in some settings. The findings highlight the potential of annotation-free SEM supervision to scale SLT across languages and reduce labeling costs, while pointing to future work on learning SEM-based visual features to close remaining gaps to gloss-based performance.

Abstract

State-of-the-art sign language translation (SLT) systems facilitate the learning process through gloss annotations, either in an end2end manner or by involving an intermediate step. Unfortunately, gloss labelled sign language data is usually not available at scale and, when available, gloss annotations widely differ from dataset to dataset. We present a novel approach using sentence embeddings of the target sentences at training time that take the role of glosses. The new kind of supervision does not need any manual annotation but it is learned on raw textual data. As our approach easily facilitates multilinguality, we evaluate it on datasets covering German (PHOENIX-2014T) and American (How2Sign) sign languages and experiment with mono- and multilingual sentence embeddings and translation systems. Our approach significantly outperforms other gloss-free approaches, setting the new state-of-the-art for data sets where glosses are not available and when no additional SLT datasets are used for pretraining, diminishing the gap between gloss-free and gloss-dependent systems.

Sign Language Translation with Sentence Embedding Supervision

TL;DR

This work tackles the reliance on scarce gloss annotations in sign language translation by introducing sentence embedding supervision (SEM). It decomposes SLT into sign2sem and sem2text modules, enabling gloss-free and multilingual translation through pretraining on raw text and end-to-end fine-tuning, with two end-to-end variants: sign2sem2text and sign2(sem+text). Empirical results on PHOENIX-2014T (DGS) and How2Sign (ASL) show that SEM-based approaches, especially the multitask variant with a pretrained text decoder (sign2(sem+text) with mBART), achieve state-of-the-art gloss-free performance on How2Sign and surpass prior gloss-free baselines on PHOENIX-2014T, though they still lag behind gloss-based systems in some settings. The findings highlight the potential of annotation-free SEM supervision to scale SLT across languages and reduce labeling costs, while pointing to future work on learning SEM-based visual features to close remaining gaps to gloss-based performance.

Abstract

State-of-the-art sign language translation (SLT) systems facilitate the learning process through gloss annotations, either in an end2end manner or by involving an intermediate step. Unfortunately, gloss labelled sign language data is usually not available at scale and, when available, gloss annotations widely differ from dataset to dataset. We present a novel approach using sentence embeddings of the target sentences at training time that take the role of glosses. The new kind of supervision does not need any manual annotation but it is learned on raw textual data. As our approach easily facilitates multilinguality, we evaluate it on datasets covering German (PHOENIX-2014T) and American (How2Sign) sign languages and experiment with mono- and multilingual sentence embeddings and translation systems. Our approach significantly outperforms other gloss-free approaches, setting the new state-of-the-art for data sets where glosses are not available and when no additional SLT datasets are used for pretraining, diminishing the gap between gloss-free and gloss-dependent systems.
Paper Structure (22 sections, 2 equations, 5 figures, 6 tables)

This paper contains 22 sections, 2 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. SEM-based Architectures
  4. $\bullet$ sign2sem Module
  5. Pretraining
  6. $\bullet$ sem2text Module
  7. Pretraining
  8. $\bullet$ sign2sem2text
  9. $\bullet$ sign2(sem+text)
  10. Experimental Settings
  11. RWTH-PHOENIX-2014T camgoz18
  12. How2Sign duarteEtAl:2021
  13. Results and Discussion
  14. Reconstruction quality: sem2text.
  15. SEM-based vs gloss-based SLT.
  16. ...and 7 more sections

Figures (5)

  • Figure 1: sign2sem and sem2text independent modules for the SLT task (left plot). End2end architectures: pipeline system sign2sem2text and multitask system sign2(sem+text) (right plot).
  • Figure 2: Average BLEURT score on different token length intervals on PHOENIX-2014T and How2Sign test.
  • Figure 3: Validation BLEU on PHOENIX without supervision (top plot), with gloss supervision (middle plot) and with SEM supervision (bottom plot).
  • Figure 4: Variation of the average BLEU score on different token length intervals on PHOENIX-2014T (top) and How2Sign (bottom) test sets.
  • Figure 5: Variation of the average chrF score on different token length intervals on PHOENIX-2014T (top) and How2Sign (bottom) test sets.