An Efficient Sign Language Translation Using Spatial Configuration and Motion Dynamics with LLMs

TL;DR

This work tackles gloss-free Sign Language Translation (SLT) by reducing reliance on domain-specific visual encoder fine-tuning. It introduces SpaMo, a framework that separately extracts spatial configurations with a frozen ViT (via $S^2$ scaling) and motion dynamics with a frozen VideoMAE, fuses them through a Sign Adapter, and translates with a LoRA-tuned LLM, aided by a Visual-Text Alignment warm-up. SpaMo achieves state-of-the-art results on PHOENIX14T, CSL-Daily, and How2Sign, supported by comprehensive ablations and qualitative analyses that illuminate how the LLM interprets sign video embeddings. The approach offers a practical, data-efficient path for SLT, leveraging off-the-shelf encoders and light alignment to narrow the modality gap while avoiding heavy, domain-specific fine-tuning.

Abstract

Gloss-free Sign Language Translation (SLT) converts sign videos directly into spoken language sentences without relying on glosses. Recently, Large Language Models (LLMs) have shown remarkable translation performance in gloss-free methods by harnessing their powerful natural language generation capabilities. However, these methods often rely on domain-specific fine-tuning of visual encoders to achieve optimal results. By contrast, this paper emphasizes the importance of capturing the spatial configurations and motion dynamics inherent in sign language. With this in mind, we introduce Spatial and Motion-based Sign Language Translation (SpaMo), a novel LLM-based SLT framework. The core idea of SpaMo is simple yet effective. We first extract spatial and motion features using off-the-shelf visual encoders and then input these features into an LLM with a language prompt. Additionally, we employ a visual-text alignment process as a warm-up before the SLT supervision. Our experiments demonstrate that SpaMo achieves state-of-the-art performance on two popular datasets, PHOENIX14T and How2Sign.

Figures (6)

• Figure 1: Visual examples of spatial configurations and motion dynamics in sign language. The images are sourced from WLASL li2020word.
• Figure 2: An overview of the SpaMo framework, which consists of three parts: (i) Sign Feature Extraction: Spatial and motion features are extracted using SE and ME, using the $S^2$ and sliding window approaches to capture detailed spatial configurations and motion dynamics. (ii) VT-Align: The extracted features are combined within SA to form a unified sign feature. During training, a warm-up process is employed to ensure that SA has well-initialized weights, effectively bridging the modality gap between the sign video and text. (iii) LLM: the LLM processes the sign feature along with a language-instructive prompt and is trained using LoRA.
• Figure 3: An overview of Sign Adapter.
• Figure 4: Ablation study for SE and ME. $S^2$ represents Scaling on Scales, and $s$ denotes stride size. Note that the presented results do not include VT-Align.
• Figure 5: The t-SNE visualization of sign features. Different colors represent features with distinct semantics, while gray points are other categories not listed.