LLaVA-SLT: Visual Language Tuning for Sign Language Translation

TL;DR

LLaVA-SLT introduces a scalable, gloss-free SLT framework that combines linguistic continued pretraining of large language models, visual contrastive pretraining of a hierarchical visual encoder, and visual language tuning with a lightweight MLP connector to align visual-language embeddings with LLM tokens. The approach is validated on CSL-Daily and Phoenix-2014T, achieving state-of-the-art gloss-free SLT results and narrowing the gap to gloss-based methods, especially when leveraging extra annotation-free CSL-400h data. Ablation studies demonstrate the importance of model scale, data, and the word-level visual language representations, as well as the effectiveness of prompting strategies for guiding LLM outputs. The work highlights a practical path toward scalable, data-efficient sign language translation using large multimodal models, with potential social impact through improved communication accessibility.

Abstract

In the realm of Sign Language Translation (SLT), reliance on costly gloss-annotated datasets has posed a significant barrier. Recent advancements in gloss-free SLT methods have shown promise, yet they often largely lag behind gloss-based approaches in terms of translation accuracy. To narrow this performance gap, we introduce LLaVA-SLT, a pioneering Large Multimodal Model (LMM) framework designed to leverage the power of Large Language Models (LLMs) through effectively learned visual language embeddings. Our model is trained through a trilogy. First, we propose linguistic continued pretraining. We scale up the LLM and adapt it to the sign language domain using an extensive corpus dataset, effectively enhancing its textual linguistic knowledge about sign language. Then, we adopt visual contrastive pretraining to align the visual encoder with a large-scale pretrained text encoder. We propose hierarchical visual encoder that learns a robust word-level intermediate representation that is compatible with LLM token embeddings. Finally, we propose visual language tuning. We freeze pretrained models and employ a lightweight trainable MLP connector. It efficiently maps the pretrained visual language embeddings into the LLM token embedding space, enabling downstream SLT task. Our comprehensive experiments demonstrate that LLaVA-SLT outperforms the state-of-the-art methods. By using extra annotation-free data, it even closes to the gloss-based accuracy.

Figures (5)

• Figure 1: LLaVA-SLT is a highly scalable LMM framework tamed for sign language translation, capable of boosting the performance with expanding multimodal data.
• Figure 2: Method overview. We train LLaVA-SLT in three stages. First, we collect sign language corpus and scale up the LLMs to enhance the linguistc capabilities in sign language (Sec. \ref{['sec:3.1']}). Subsequently, we employ a large-scale pretrained text encoder to supervise the visual encoder using both inner and outer contrastive losses (Sec. \ref{['sec:3.2']}). Finally, we connect the pretrained visual and language models via a lightweight MLP connector, equipped with an effective prompting strategy, which efficiently enables downstream SLT task (Sec. \ref{['sec:3.3']}).
• Figure 3: Wordcloud illustration of the CSL-Corpus dataset.
• Figure 4: Hierarchical visual architecture. The visual representation is explicitly separated into three levels: frame level, word level, and sentence level.
• Figure A: CSL-400h dataset gallery. We showcase several sign videos and their corresponding source texts in the dataset.