Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Sign Language Models: Toward 3D American Sign Language Translation

Sen Zhang, Xiaoxiao He, Di Liu, Zhaoyang Xia, Mingyu Zhao, Chaowei Tan, Vivian Li, Bo Liu, Dimitris N. Metaxas, Mubbasir Kapadia

TL;DR

This work addresses the challenge of translating 3D American Sign Language in virtual environments by grounding an LLM backbone with environment-robust 3D gesture representations. It introduces a three-stage pipeline: a VQ-VAE-based 3D sign language tokenizer over SMPL-X motions, modality alignment to an LLM, and instruction tuning to enable flexible translation. The approach demonstrates direct gesture-to-text translation and instruction-guided translation, outperforming 2D, motion-to-text baselines and showing the benefits of explicit gesture-to-embedding alignment. This framework advances embodied multimodal understanding in LLMs and holds promise for accessible, robust sign language translation in immersive and diverse real-world settings.

Abstract

We present Large Sign Language Models (LSLM), a novel framework for translating 3D American Sign Language (ASL) by leveraging Large Language Models (LLMs) as the backbone, which can benefit hearing-impaired individuals' virtual communication. Unlike existing sign language recognition methods that rely on 2D video, our approach directly utilizes 3D sign language data to capture rich spatial, gestural, and depth information in 3D scenes. This enables more accurate and resilient translation, enhancing digital communication accessibility for the hearing-impaired community. Beyond the task of ASL translation, our work explores the integration of complex, embodied multimodal languages into the processing capabilities of LLMs, moving beyond purely text-based inputs to broaden their understanding of human communication. We investigate both direct translation from 3D gesture features to text and an instruction-guided setting where translations can be modulated by external prompts, offering greater flexibility. This work provides a foundational step toward inclusive, multimodal intelligent systems capable of understanding diverse forms of language.

Large Sign Language Models: Toward 3D American Sign Language Translation

TL;DR

This work addresses the challenge of translating 3D American Sign Language in virtual environments by grounding an LLM backbone with environment-robust 3D gesture representations. It introduces a three-stage pipeline: a VQ-VAE-based 3D sign language tokenizer over SMPL-X motions, modality alignment to an LLM, and instruction tuning to enable flexible translation. The approach demonstrates direct gesture-to-text translation and instruction-guided translation, outperforming 2D, motion-to-text baselines and showing the benefits of explicit gesture-to-embedding alignment. This framework advances embodied multimodal understanding in LLMs and holds promise for accessible, robust sign language translation in immersive and diverse real-world settings.

Abstract

We present Large Sign Language Models (LSLM), a novel framework for translating 3D American Sign Language (ASL) by leveraging Large Language Models (LLMs) as the backbone, which can benefit hearing-impaired individuals' virtual communication. Unlike existing sign language recognition methods that rely on 2D video, our approach directly utilizes 3D sign language data to capture rich spatial, gestural, and depth information in 3D scenes. This enables more accurate and resilient translation, enhancing digital communication accessibility for the hearing-impaired community. Beyond the task of ASL translation, our work explores the integration of complex, embodied multimodal languages into the processing capabilities of LLMs, moving beyond purely text-based inputs to broaden their understanding of human communication. We investigate both direct translation from 3D gesture features to text and an instruction-guided setting where translations can be modulated by external prompts, offering greater flexibility. This work provides a foundational step toward inclusive, multimodal intelligent systems capable of understanding diverse forms of language.

Paper Structure

This paper contains 23 sections, 2 equations, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Sign Language Recognition
  4. Sign Language in Large Language Models
  5. LLM-based Motion Understanding
  6. Methods
  7. Environment Agnostic 3D Sign Language Tokenization
  8. Sign Language Motion to Text Alignment
  9. Training Scheme
  10. Experiments
  11. Experimental Setup
  12. Comparison with Motion-to-Text Approaches
  13. Modality Alignment and Pretraining
  14. Instruction Finetuning
  15. Ablation Study on LLM Backbone
  16. ...and 8 more sections

Figures (4)

  • Figure 1: The training pipeline for our Large Sign Language Model (LSLM) initiates with a pretrained text-based large language model and proceeds in three stages. First, we train a motion vector-quantized variational autoencoder (VQ-VAE); its quantized outputs are then used with a projection layer for gesture-text alignment. Second, these aligned gesture features serve as input to pretrain the LLM, enabling foundational sign language understanding. Finally, the model is instruction-finetuned to develop its ability to follow instructions with sign language.
  • Figure 2: Evaluation of VQ-VAE codebook sizes on the How2Sign 3D dataset. A codebook size of 1024 offers the best trade-off, achieving the lowest FID while maintaining competitive MPJPE and PAMPJPE scores.
  • Figure 3: Instruction-guided Sign Language Recognition (SLR) examples from our LSLM framework. Each case shows the model's translation given a gesture sequence and prompt, alongside the ground truth. Outputs may slightly differ in wording but preserve the core meaning.
  • Figure 4: Instruction-guided Sign Language Recognition (SLR) examples from our LSLM framework with Qwen backbone. Each case shows the model's translation given a gesture sequence and prompt, alongside the ground truth. Outputs may slightly differ in wording but preserve the core meaning.