Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Influence of Iconicity in Transfer Learning for Sign Language Recognition

Keren Artiaga, Conor Lynch, Haithem Afli, Mohammed Hasanuzzaman

TL;DR

This body of work examines the necessity of likenesses on effective knowledge transfer by comparing TL performance between iconic signs of two different sign language pairs: Chinese to Arabic and Greek to Flemish.

Abstract

Most sign language recognition research relies on Transfer Learning (TL) from vision-based datasets such as ImageNet. Some extend this to alternatively available language datasets, often focusing on signs with cross-linguistic similarities. This body of work examines the necessity of these likenesses on effective knowledge transfer by comparing TL performance between iconic signs of two different sign language pairs: Chinese to Arabic and Greek to Flemish. Google Mediapipe was utilised as an input feature extractor, enabling spatial information of these signs to be processed with a Multilayer Perceptron architecture and the temporal information with a Gated Recurrent Unit. Experimental results showed a 7.02% improvement for Arabic and 1.07% for Flemish when conducting iconic TL from Chinese and Greek respectively.

The Influence of Iconicity in Transfer Learning for Sign Language Recognition

TL;DR

This body of work examines the necessity of likenesses on effective knowledge transfer by comparing TL performance between iconic signs of two different sign language pairs: Chinese to Arabic and Greek to Flemish.

Abstract

Most sign language recognition research relies on Transfer Learning (TL) from vision-based datasets such as ImageNet. Some extend this to alternatively available language datasets, often focusing on signs with cross-linguistic similarities. This body of work examines the necessity of these likenesses on effective knowledge transfer by comparing TL performance between iconic signs of two different sign language pairs: Chinese to Arabic and Greek to Flemish. Google Mediapipe was utilised as an input feature extractor, enabling spatial information of these signs to be processed with a Multilayer Perceptron architecture and the temporal information with a Gated Recurrent Unit. Experimental results showed a 7.02% improvement for Arabic and 1.07% for Flemish when conducting iconic TL from Chinese and Greek respectively.
Paper Structure (12 sections, 5 equations, 5 figures, 9 tables)

This paper contains 12 sections, 5 equations, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Data Pre-processing and Extracting MediaPipe Keypoints
  5. MLP-GRU
  6. Transfer Learning (TL) and Results
  7. Ablation Study
  8. Effect of pre-training with non-iconic signs
  9. Effect of pre-training with iconic+non-iconic signs
  10. Effect of fewer similarities in iconic signs
  11. Effect of pre-training with ImageNet
  12. Discussion and Conclusion

Figures (5)

  • Figure 1: Sign gesture hand activity for individual concepts across languages. visualiconicity
  • Figure 2: Sign gesture hand activity for groups of concepts across languages. visualiconicity
  • Figure 3: MediaPipe keypoint landmarks detailing the anatomy sign for Skull (KArSL) and Head (CSL)
  • Figure 4: MLP-GRU execution on SLR TL tasks.
  • Figure 5: MLP-GRU execution on SLR TL tasks.