Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

sign.mt: Real-Time Multilingual Sign Language Translation Application

Amit Moryossef

TL;DR

sign.mt tackles the communication barrier between deaf or hard-of-hearing users and speakers by delivering real-time multilingual translation between spoken and signed languages. The work presents a modular, end-to-end architecture in which spoken-to-signed and signed-to-spoken translations share core infrastructure and can be extended with new models, all running on-device with offline support. Key contributions include SignWriting-based representations, multiple rendering options for sign animation, and an open invitation for community-driven model integration under CC BY-NC-SA 4.0. The project demonstrates practical impact through a public web app with broad localization and clear pathway for research-to-implementation collaboration.

Abstract

This demo paper presents sign.mt, an open-source application pioneering real-time multilingual bi-directional translation between spoken and signed languages. Harnessing state-of-the-art open-source models, this tool aims to address the communication divide between the hearing and the deaf, facilitating seamless translation in both spoken-to-signed and signed-to-spoken translation directions. Promising reliable and unrestricted communication, sign.mt offers offline functionality, crucial in areas with limited internet connectivity. It further enhances user engagement by offering customizable photo-realistic sign language avatars, thereby encouraging a more personalized and authentic user experience. Licensed under CC BY-NC-SA 4.0, sign.mt signifies an important stride towards open, inclusive communication. The app can be used, and modified for personal and academic uses, and even supports a translation API, fostering integration into a wider range of applications. However, it is by no means a finished product. We invite the NLP community to contribute towards the evolution of sign.mt. Whether it be the integration of more refined models, the development of innovative pipelines, or user experience improvements, your contributions can propel this project to new heights. Available at https://sign.mt, it stands as a testament to what we can achieve together, as we strive to make communication accessible to all.

sign.mt: Real-Time Multilingual Sign Language Translation Application

TL;DR

sign.mt tackles the communication barrier between deaf or hard-of-hearing users and speakers by delivering real-time multilingual translation between spoken and signed languages. The work presents a modular, end-to-end architecture in which spoken-to-signed and signed-to-spoken translations share core infrastructure and can be extended with new models, all running on-device with offline support. Key contributions include SignWriting-based representations, multiple rendering options for sign animation, and an open invitation for community-driven model integration under CC BY-NC-SA 4.0. The project demonstrates practical impact through a public web app with broad localization and clear pathway for research-to-implementation collaboration.

Abstract

This demo paper presents sign.mt, an open-source application pioneering real-time multilingual bi-directional translation between spoken and signed languages. Harnessing state-of-the-art open-source models, this tool aims to address the communication divide between the hearing and the deaf, facilitating seamless translation in both spoken-to-signed and signed-to-spoken translation directions. Promising reliable and unrestricted communication, sign.mt offers offline functionality, crucial in areas with limited internet connectivity. It further enhances user engagement by offering customizable photo-realistic sign language avatars, thereby encouraging a more personalized and authentic user experience. Licensed under CC BY-NC-SA 4.0, sign.mt signifies an important stride towards open, inclusive communication. The app can be used, and modified for personal and academic uses, and even supports a translation API, fostering integration into a wider range of applications. However, it is by no means a finished product. We invite the NLP community to contribute towards the evolution of sign.mt. Whether it be the integration of more refined models, the development of innovative pipelines, or user experience improvements, your contributions can propel this project to new heights. Available at https://sign.mt, it stands as a testament to what we can achieve together, as we strive to make communication accessible to all.
Paper Structure (6 sections, 6 figures)

This paper contains 6 sections, 6 figures.

Table of Contents

  1. Motivation
  2. Implementation
  3. Spoken-to-Signed Translation
  4. Signed-to-Spoken Translation
  5. User Engagement
  6. Distribution

Figures (6)

  • Figure 1: The Spoken-to-Signed translation pipeline.
  • Figure 2: The Signed-to-Spoken translation pipeline.
  • Figure 3: Distribution of sign.mt users across the world, over the last year.
  • Figure 4: Growth of sign.mt users over the last year.
  • Figure 5: Number of stars for the repository over time.
  • ...and 1 more figures