Spatio-temporal Sign Language Representation and Translation

TL;DR

This work tackles sign language translation under data scarcity by proposing an end-to-end spatio-temporal model that jointly learns visual representations and German text translation. The architecture fuses a 3D CNN visual encoder (ResNet3D) with a Transformer language model via a Sentence-to-Words Mapping, trained end-to-end to optimize translation directly from sign video. Despite achieving a development BLEU as high as $4.8$, the approach yields very low test performance ($0.11\pm0.06$ BLEU), highlighting severe generalization gaps and the difficulty of SLT. Analyses show reliance on high-frequency tokens and occasional <UNK> outputs, with some gains on the FocusNews subset, suggesting that more effective temporal modeling and stronger alignment between video and language are needed for robust SLT systems.

Abstract

This paper describes the DFKI-MLT submission to the WMT-SLT 2022 sign language translation (SLT) task from Swiss German Sign Language (video) into German (text). State-of-the-art techniques for SLT use a generic seq2seq architecture with customized input embeddings. Instead of word embeddings as used in textual machine translation, SLT systems use features extracted from video frames. Standard approaches often do not benefit from temporal features. In our participation, we present a system that learns spatio-temporal feature representations and translation in a single model, resulting in a real end-to-end architecture expected to better generalize to new data sets. Our best system achieved $5\pm1$ BLEU points on the development set, but the performance on the test dropped to $0.11\pm0.06$ BLEU points.

Figures (1)

• Figure 1: End-to-end architecture for sign language translation.