Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SignVLA: A Gloss-Free Vision-Language-Action Framework for Real-Time Sign Language-Guided Robotic Manipulation

Xinyu Tan, Ningwei Bai, Harry Gardener, Zhengyang Zhong, Luoyu Zhang, Liuhaichen Yang, Zhekai Duan, Monkgogi Galeitsiwe, Zezhi Tang

TL;DR

This work presents the first sign language-driven Vision-Language-Action (VLA) framework for intuitive and inclusive human-robot interaction that adopts a gloss-free paradigm and directly maps visual sign gestures to semantic instructions, enabling more natural and scalable multimodal interaction.

Abstract

We present, to our knowledge, the first sign language-driven Vision-Language-Action (VLA) framework for intuitive and inclusive human-robot interaction. Unlike conventional approaches that rely on gloss annotations as intermediate supervision, the proposed system adopts a gloss-free paradigm and directly maps visual sign gestures to semantic instructions. This design reduces annotation cost and avoids the information loss introduced by gloss representations, enabling more natural and scalable multimodal interaction. In this work, we focus on a real-time alphabet-level finger-spelling interface that provides a robust and low-latency communication channel for robotic control. Compared with large-scale continuous sign language recognition, alphabet-level interaction offers improved reliability, interpretability, and deployment feasibility in safety-critical embodied environments. The proposed pipeline transforms continuous gesture streams into coherent language commands through geometric normalization, temporal smoothing, and lexical refinement, ensuring stable and consistent interaction. Furthermore, the framework is designed to support future integration of transformer-based gloss-free sign language models, enabling scalable word-level and sentence-level semantic understanding. Experimental results demonstrate the effectiveness of the proposed system in grounding sign-derived instructions into precise robotic actions under diverse interaction scenarios. These results highlight the potential of the framework to advance accessible, scalable, and multimodal embodied intelligence.

SignVLA: A Gloss-Free Vision-Language-Action Framework for Real-Time Sign Language-Guided Robotic Manipulation

TL;DR

This work presents the first sign language-driven Vision-Language-Action (VLA) framework for intuitive and inclusive human-robot interaction that adopts a gloss-free paradigm and directly maps visual sign gestures to semantic instructions, enabling more natural and scalable multimodal interaction.

Abstract

We present, to our knowledge, the first sign language-driven Vision-Language-Action (VLA) framework for intuitive and inclusive human-robot interaction. Unlike conventional approaches that rely on gloss annotations as intermediate supervision, the proposed system adopts a gloss-free paradigm and directly maps visual sign gestures to semantic instructions. This design reduces annotation cost and avoids the information loss introduced by gloss representations, enabling more natural and scalable multimodal interaction. In this work, we focus on a real-time alphabet-level finger-spelling interface that provides a robust and low-latency communication channel for robotic control. Compared with large-scale continuous sign language recognition, alphabet-level interaction offers improved reliability, interpretability, and deployment feasibility in safety-critical embodied environments. The proposed pipeline transforms continuous gesture streams into coherent language commands through geometric normalization, temporal smoothing, and lexical refinement, ensuring stable and consistent interaction. Furthermore, the framework is designed to support future integration of transformer-based gloss-free sign language models, enabling scalable word-level and sentence-level semantic understanding. Experimental results demonstrate the effectiveness of the proposed system in grounding sign-derived instructions into precise robotic actions under diverse interaction scenarios. These results highlight the potential of the framework to advance accessible, scalable, and multimodal embodied intelligence.
Paper Structure (21 sections, 1 equation, 2 figures, 3 tables)

This paper contains 21 sections, 1 equation, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Sign-to-Word: Alphabet-Level Perception Buffer
  4. Data Augmentation and Dataset Expansion
  5. Feature Extraction and Modeling
  6. Linguistic Buffering and Instruction Synthesis
  7. VLA Policy and Task Execution
  8. Model and Training
  9. Sign Language Model
  10. Description
  11. Model Training
  12. VLA Model
  13. Experiments
  14. Experimental Setup
  15. Spatial Calibration and Scaling
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Qualitative demonstration of the Sign-VLA policy across three distinct tasks. The sequences show the robot successfully executing instructions for (a) color-specific objects, (b) localized target areas, and (c) basic geometric shapes.
  • Figure 2: Illustration of a potential future extension of our framework using a transformer-based gloss-free sign language model. The encoder extracts spatial and temporal features from continuous sign videos, while the decoder generates natural language instructions that can be directly grounded by the VLA policy yang2024signformerneededgeai.