A Comparative Study of Continuous Sign Language Recognition Techniques

TL;DR

An empirical evaluation of recent deep learning CSLR techniques and assess their performance across various datasets and sign languages to determine their efficacy in modeling different sign languages.

Abstract

Continuous Sign Language Recognition (CSLR) focuses on the interpretation of a sequence of sign language gestures performed continually without pauses. In this study, we conduct an empirical evaluation of recent deep learning CSLR techniques and assess their performance across various datasets and sign languages. The models selected for analysis implement a range of approaches for extracting meaningful features and employ distinct training strategies. To determine their efficacy in modeling different sign languages, these models were evaluated using multiple datasets, specifically RWTH-PHOENIX-Weather-2014, ArabSign, and GrSL, each representing a unique sign language. The performance of the models was further tested with unseen signers and sentences. The conducted experiments establish new benchmarks on the selected datasets and provide valuable insights into the robustness and generalization of the evaluated techniques under challenging scenarios.

Figures (7)

• Figure 1: The general framework of CSLR.
• Figure 2: Samples from the utilized datasets Phoenix2014 (top), ArabSign (middle), and GrSL (bottom).
• Figure 3: WERs of the evaluated models in different evaluation settings using (a) Phoenix2014, (b) ArabSign, and (c) GrSL datasets.
• Figure 4: Samples of gloss predictions of evaluated models with (a) Signer-Dep, (b) Signer-Indep, and (c) Unseen-Sent evaluation settings. Errors are colored in pink, where (D) indicates a deletion error, (I) an insertion error, and (S) a substitution error.
• Figure 5: GradCam selvaraju2017grad visualization of gloss "KOMMEN" by the evaluated models on Phoenix2014 Signer-Dep. The red areas indicate the most attended to regions.