Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The American Sign Language Knowledge Graph: Infusing ASL Models with Linguistic Knowledge

Lee Kezar, Nidhi Munikote, Zian Zeng, Zed Sehyr, Naomi Caselli, Jesse Thomason

TL;DR

The ASLKG is introduced, compiled from twelve sources of expert linguistic knowledge, to train neuro-symbolic models for 3 ASL understanding tasks, achieving accuracies of 91% on ISR, 14% for predicting the semantic features of unseen signs, and 36% for classifying the topic of Youtube-ASL videos.

Abstract

Language models for American Sign Language (ASL) could make language technologies substantially more accessible to those who sign. To train models on tasks such as isolated sign recognition (ISR) and ASL-to-English translation, datasets provide annotated video examples of ASL signs. To facilitate the generalizability and explainability of these models, we introduce the American Sign Language Knowledge Graph (ASLKG), compiled from twelve sources of expert linguistic knowledge. We use the ASLKG to train neuro-symbolic models for 3 ASL understanding tasks, achieving accuracies of 91% on ISR, 14% for predicting the semantic features of unseen signs, and 36% for classifying the topic of Youtube-ASL videos.

The American Sign Language Knowledge Graph: Infusing ASL Models with Linguistic Knowledge

TL;DR

The ASLKG is introduced, compiled from twelve sources of expert linguistic knowledge, to train neuro-symbolic models for 3 ASL understanding tasks, achieving accuracies of 91% on ISR, 14% for predicting the semantic features of unseen signs, and 36% for classifying the topic of Youtube-ASL videos.

Abstract

Language models for American Sign Language (ASL) could make language technologies substantially more accessible to those who sign. To train models on tasks such as isolated sign recognition (ISR) and ASL-to-English translation, datasets provide annotated video examples of ASL signs. To facilitate the generalizability and explainability of these models, we introduce the American Sign Language Knowledge Graph (ASLKG), compiled from twelve sources of expert linguistic knowledge. We use the ASLKG to train neuro-symbolic models for 3 ASL understanding tasks, achieving accuracies of 91% on ISR, 14% for predicting the semantic features of unseen signs, and 36% for classifying the topic of Youtube-ASL videos.

Paper Structure

This paper contains 42 sections, 8 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Knowledge-Infused Learning
  4. ASL Linguistics
  5. Phonology
  6. Lexical Semantics
  7. The ASL Knowledge Graph
  8. Definition
  9. ASLKG Statistics
  10. Partitioning Facts for ML Tasks
  11. Tasks Facilitated by the ASLKG
  12. Intrinsic Fact Verification
  13. Grounding ASL Videos to Phonemes
  14. Inferring Signs and their Meanings
  15. Isolated Sign Recognition (ISR).
  16. ...and 27 more sections

Figures (2)

  • Figure 1: The ASLKG relates the form (e.g., 2/V handshape) and meaning (e.g., related to sight) of signs in the ASL lexicon. We use this knowledge to neuro-symbolically recognize signs (e.g., read) and infer their meaning.
  • Figure :