Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sign Language Sense Disambiguation

Jana Grimm, Miriam Winkler, Oliver Kraus, Tanalp Agustoslu

TL;DR

This project explores methods to enhance sign language translation of German sign language, specifically focusing on disambiguation of homonyms by training transformer-based models on various bodypart representations to shift the focus on said bodypart.

Abstract

This project explores methods to enhance sign language translation of German sign language, specifically focusing on disambiguation of homonyms. Sign language is ambiguous and understudied which is the basis for our experiments. We approach the improvement by training transformer-based models on various bodypart representations to shift the focus on said bodypart. To determine the impact of, e.g., the hand or mouth representations, we experiment with different combinations. The results show that focusing on the mouth increases the performance in small dataset settings while shifting the focus on the hands retrieves better results in larger dataset settings. Our results contribute to better accessibility for non-hearing persons by improving the systems powering digital assistants, enabling a more accurate interaction. The code for this project can be found on GitHub.

Sign Language Sense Disambiguation

TL;DR

This project explores methods to enhance sign language translation of German sign language, specifically focusing on disambiguation of homonyms by training transformer-based models on various bodypart representations to shift the focus on said bodypart.

Abstract

This project explores methods to enhance sign language translation of German sign language, specifically focusing on disambiguation of homonyms. Sign language is ambiguous and understudied which is the basis for our experiments. We approach the improvement by training transformer-based models on various bodypart representations to shift the focus on said bodypart. To determine the impact of, e.g., the hand or mouth representations, we experiment with different combinations. The results show that focusing on the mouth increases the performance in small dataset settings while shifting the focus on the hands retrieves better results in larger dataset settings. Our results contribute to better accessibility for non-hearing persons by improving the systems powering digital assistants, enabling a more accurate interaction. The code for this project can be found on GitHub.
Paper Structure (17 sections, 3 figures, 4 tables)

This paper contains 17 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Sign Language Tasks
  4. Sign Language Transformers
  5. Sign Language Resources
  6. Methods
  7. Dataset
  8. Data Preprocessinng
  9. Baseline Data Preparation
  10. Body Part Extraction
  11. Signjoey - Sign Language Transformer
  12. Evaluation Results
  13. Data and Model Variations
  14. Integration of body part representations
  15. Homonym analysis
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Example for body part extraction.
  • Figure 2: One example of the matching problem in the RWTH-PHOENIX-Weather Database of German Sign Language Koller2015Continuous. The gloss on the top is the original gloss aligned with the video, the glosses on the bottom are possible matching glosses aligned with the textual translation of the video.
  • Figure 3: One example of the translation of our best-performing model. The ambiguous sign "tomorrow" ("morgen") is translated as "day" ("tag") incorrectly.