POSESTITCH-SLT: Linguistically Inspired Pose-Stitching for End-to-End Sign Language Translation

TL;DR

PoseStitch-SLT tackles data scarcity and signer privacy in gloss-free sign language translation by generating large-scale synthetic pose–sentence pairs through linguistically informed templates and pose stitching. It leverages a standard Transformer encoder–decoder pretrained on BLiMP/BPCC-derived data aligned to CISLR and WLASL vocabularies, with a linear annealing schedule blending synthetic and real data. Empirical results on How2Sign and iSign show substantial BLEU-4 gains over prior methods, demonstrating that template-driven synthetic supervision can effectively compensate for limited real data in low-resource SLT. This approach offers a scalable, privacy-preserving path to improving gloss-free SLT and motivates future work on expanding vocabulary breadth and incorporating sign-language grammar cues across more languages.

Abstract

Sign language translation remains a challenging task due to the scarcity of large-scale, sentence-aligned datasets. Prior arts have focused on various feature extraction and architectural changes to support neural machine translation for sign languages. We propose POSESTITCH-SLT, a novel pre-training scheme that is inspired by linguistic-templates-based sentence generation technique. With translation comparison on two sign language datasets, How2Sign and iSign, we show that a simple transformer-based encoder-decoder architecture outperforms the prior art when considering template-generated sentence pairs in training. We achieve BLEU-4 score improvements from 1.97 to 4.56 on How2Sign and from 0.55 to 3.43 on iSign, surpassing prior state-of-the-art methods for pose-based gloss-free translation. The results demonstrate the effectiveness of template-driven synthetic supervision in low-resource sign language settings.

Figures (19)

• Figure 1: The figure shows the PoseStitch-SLT pipeline for generating a Sign Language Dataset for translation using pose stitching based on linguistic templates. Starting with a common vocabulary shared across datasets like CISLR, BLIMP, and WLASL, sentences are generated using BLIMP linguistic templates warstadt-etal-2020-blimp-benchmark. For each generated sentence, corresponding sign language poses for words/gloss are stitched together. Frames are extracted from the stitched pose video to form a sequence of keypoints. These keypoints extracted from the face, hands, and body are concatenated to create pose vectors. This sequence of pose vectors serves as input to an encoder-decoder-based transformer model. The sentence example is taken from the How2Sign dataset, saying, “I hope you're having fun”.
• Figure 2: (a) From the filtered raw BPCC Corpus, sentences are matched with CISLR and WLASL vocabulary, and the sentence length distribution is aligned with iSign and How2Sign. Two datasets, BPCC-ISL and BPCC-ASL are created after pose stitching (b). Using templates from BLIMP warstadt-etal-2020-blimp-benchmark and the common vocabulary of BLIMP, CISLR, and WLASL, two additional datasets are created: BLIMP-ISL and BLIMP-ASL after pose stitching.
• Figure 3: Sentence length distribution between iSign train and BPCC-ISL dataset before merging sentences
• Figure 4: Sentence length distribution between iSign train and BPCC-ISL dataset after merging sentences
• Figure 5: Sentence length distribution between How2Sign and BPCC-ASL dataset before merging sentences