Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-Attention Based Influence Model for Manual and Nonmanual Sign Language Analysis

Lipisha Chaudhary, Fei Xu, Ifeoma Nwogu

TL;DR

This work examines and reports on the extent to which facial expressions contribute to understanding sign language phrases, and proposes a new parallel cross-attention decoding mechanism that is useful for quantifying the influence of each input modality on the output.

Abstract

Both manual (relating to the use of hands) and non-manual markers (NMM), such as facial expressions or mouthing cues, are important for providing the complete meaning of phrases in American Sign Language (ASL). Efforts have been made in advancing sign language to spoken/written language understanding, but most of these have primarily focused on manual features only. In this work, using advanced neural machine translation methods, we examine and report on the extent to which facial expressions contribute to understanding sign language phrases. We present a sign language translation architecture consisting of two-stream encoders, with one encoder handling the face and the other handling the upper body (with hands). We propose a new parallel cross-attention decoding mechanism that is useful for quantifying the influence of each input modality on the output. The two streams from the encoder are directed simultaneously to different attention stacks in the decoder. Examining the properties of the parallel cross-attention weights allows us to analyze the importance of facial markers compared to body and hand features during a translating task.

Cross-Attention Based Influence Model for Manual and Nonmanual Sign Language Analysis

TL;DR

This work examines and reports on the extent to which facial expressions contribute to understanding sign language phrases, and proposes a new parallel cross-attention decoding mechanism that is useful for quantifying the influence of each input modality on the output.

Abstract

Both manual (relating to the use of hands) and non-manual markers (NMM), such as facial expressions or mouthing cues, are important for providing the complete meaning of phrases in American Sign Language (ASL). Efforts have been made in advancing sign language to spoken/written language understanding, but most of these have primarily focused on manual features only. In this work, using advanced neural machine translation methods, we examine and report on the extent to which facial expressions contribute to understanding sign language phrases. We present a sign language translation architecture consisting of two-stream encoders, with one encoder handling the face and the other handling the upper body (with hands). We propose a new parallel cross-attention decoding mechanism that is useful for quantifying the influence of each input modality on the output. The two streams from the encoder are directed simultaneously to different attention stacks in the decoder. Examining the properties of the parallel cross-attention weights allows us to analyze the importance of facial markers compared to body and hand features during a translating task.
Paper Structure (22 sections, 6 equations, 6 figures, 2 tables)

This paper contains 22 sections, 6 equations, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multi-Modal Sign Language Translation
  4. Manual and Non-Manual Markers
  5. Method
  6. Parallel Cross-Attention Decoder Transformer
  7. Two-stream encoding pipeline
  8. Decoding pipeline
  9. Implementation and Evaluation Details
  10. Datasets
  11. 2-stage Input Feature Extraction Process
  12. 3D keypoints
  13. Rotation Matrix
  14. Width
  15. Training and Network Details
  16. ...and 7 more sections

Figures (6)

  • Figure 1: Overview of the proposed architecture showing the two-stream dual encoder and the dual-cross-attention based decoder.
  • Figure 2: Face crop samples from the datasets. Top: from the Phoenix2014 dataset; Bottom: from the ASLing dataset
  • Figure 3: The $(x, y)$ joint plots for body (left) and face (right). Note that we use 3D points $(x, y, z)$, in our analysis.
  • Figure 4: Overview of the 2-stage input feature extraction process.
  • Figure 5: Learned attention weights for Phoenix2014T dataset. Top: attention weights based on the input manual markers (body features); bottom: attention weights based on the input non-manual markers (facial features)
  • ...and 1 more figures