Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How Do Lexical Senses Correspond Between Spoken German and German Sign Language?

Melis Çelikkol, Wei Zhao

Abstract

Sign language lexicographers construct bilingual dictionaries by establishing word-to-sign mappings, where polysemous and homonymous words corresponding to different signs across contexts are often underrepresented. A usage-based approach examining how word senses map to signs can identify such novel mappings absent from current dictionaries, enriching lexicographic resources. We address this by analyzing German and German Sign Language (Deutsche Gebärdensprache, DGS), manually annotating 1,404 word use-to-sign ID mappings derived from 32 words from the German Word Usage Graph (D-WUG) and 49 signs from the Digital Dictionary of German Sign Language (DW-DGS). We identify three correspondence types: Type 1 (one-to-many), Type 2 (many-to-one), and Type 3 (one-to-one), plus No Match cases. We evaluate computational methods: Exact Match (EM) and Semantic Similarity (SS) using SBERT embeddings. SS substantially outperforms EM overall 88.52% vs. 71.31%), with dramatic gains for Type 1 (+52.1 pp). Our work establishes the first annotated dataset for cross-modal sense correspondence and reveals which correspondence patterns are computationally identifiable. Our code and dataset are made publicly available.

How Do Lexical Senses Correspond Between Spoken German and German Sign Language?

Abstract

Sign language lexicographers construct bilingual dictionaries by establishing word-to-sign mappings, where polysemous and homonymous words corresponding to different signs across contexts are often underrepresented. A usage-based approach examining how word senses map to signs can identify such novel mappings absent from current dictionaries, enriching lexicographic resources. We address this by analyzing German and German Sign Language (Deutsche Gebärdensprache, DGS), manually annotating 1,404 word use-to-sign ID mappings derived from 32 words from the German Word Usage Graph (D-WUG) and 49 signs from the Digital Dictionary of German Sign Language (DW-DGS). We identify three correspondence types: Type 1 (one-to-many), Type 2 (many-to-one), and Type 3 (one-to-one), plus No Match cases. We evaluate computational methods: Exact Match (EM) and Semantic Similarity (SS) using SBERT embeddings. SS substantially outperforms EM overall 88.52% vs. 71.31%), with dramatic gains for Type 1 (+52.1 pp). Our work establishes the first annotated dataset for cross-modal sense correspondence and reveals which correspondence patterns are computationally identifiable. Our code and dataset are made publicly available.
Paper Structure (39 sections, 2 equations, 1 figure, 16 tables)

This paper contains 39 sections, 2 equations, 1 figure, 16 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Lexical Ambiguities
  4. Computational Approaches to Sign Languages.
  5. Research Gap.
  6. Our Dataset
  7. Data Sources
  8. D-WUG
  9. DW-DGS
  10. Video IDs Instead of Glosses.
  11. Human Annotation
  12. Type 1 (One-to-Many).
  13. Type 2 (Many-to-One).
  14. Type 3 (One-to-One).
  15. No Match.
  16. ...and 24 more sections

Figures (1)

  • Figure 1: Confusion matrix visualizing agreement between model predictions and human annotations across 20 test words, clearly showing the model's complete inability to identify Type 2 (many-to-one) correspondence, with all 5 Type 2 cases misclassified as either No Match or Type 3.