Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MM-WLAuslan: Multi-View Multi-Modal Word-Level Australian Sign Language Recognition Dataset

Xin Shen, Heming Du, Hongwei Sheng, Shuyun Wang, Hui Chen, Huiqiang Chen, Zhuojie Wu, Xiaobiao Du, Jiaying Ying, Ruihan Lu, Qingzheng Xu, Xin Yu

TL;DR

This work curates the first large-scale Multi-view Multi-modal Word-Level Australian Sign Language recognition dataset, dubbed MM-WLAuslan, and benchmarks results indicate that MM-WLAuslan is a challenging ISLR dataset, and the hope this dataset will contribute to the development of Auslan and the advancement of sign languages worldwide.

Abstract

Isolated Sign Language Recognition (ISLR) focuses on identifying individual sign language glosses. Considering the diversity of sign languages across geographical regions, developing region-specific ISLR datasets is crucial for supporting communication and research. Auslan, as a sign language specific to Australia, still lacks a dedicated large-scale word-level dataset for the ISLR task. To fill this gap, we curate \underline{\textbf{the first}} large-scale Multi-view Multi-modal Word-Level Australian Sign Language recognition dataset, dubbed MM-WLAuslan. Compared to other publicly available datasets, MM-WLAuslan exhibits three significant advantages: (1) the largest amount of data, (2) the most extensive vocabulary, and (3) the most diverse of multi-modal camera views. Specifically, we record 282K+ sign videos covering 3,215 commonly used Auslan glosses presented by 73 signers in a studio environment. Moreover, our filming system includes two different types of cameras, i.e., three Kinect-V2 cameras and a RealSense camera. We position cameras hemispherically around the front half of the model and simultaneously record videos using all four cameras. Furthermore, we benchmark results with state-of-the-art methods for various multi-modal ISLR settings on MM-WLAuslan, including multi-view, cross-camera, and cross-view. Experiment results indicate that MM-WLAuslan is a challenging ISLR dataset, and we hope this dataset will contribute to the development of Auslan and the advancement of sign languages worldwide. All datasets and benchmarks are available at MM-WLAuslan.

MM-WLAuslan: Multi-View Multi-Modal Word-Level Australian Sign Language Recognition Dataset

TL;DR

This work curates the first large-scale Multi-view Multi-modal Word-Level Australian Sign Language recognition dataset, dubbed MM-WLAuslan, and benchmarks results indicate that MM-WLAuslan is a challenging ISLR dataset, and the hope this dataset will contribute to the development of Auslan and the advancement of sign languages worldwide.

Abstract

Isolated Sign Language Recognition (ISLR) focuses on identifying individual sign language glosses. Considering the diversity of sign languages across geographical regions, developing region-specific ISLR datasets is crucial for supporting communication and research. Auslan, as a sign language specific to Australia, still lacks a dedicated large-scale word-level dataset for the ISLR task. To fill this gap, we curate \underline{\textbf{the first}} large-scale Multi-view Multi-modal Word-Level Australian Sign Language recognition dataset, dubbed MM-WLAuslan. Compared to other publicly available datasets, MM-WLAuslan exhibits three significant advantages: (1) the largest amount of data, (2) the most extensive vocabulary, and (3) the most diverse of multi-modal camera views. Specifically, we record 282K+ sign videos covering 3,215 commonly used Auslan glosses presented by 73 signers in a studio environment. Moreover, our filming system includes two different types of cameras, i.e., three Kinect-V2 cameras and a RealSense camera. We position cameras hemispherically around the front half of the model and simultaneously record videos using all four cameras. Furthermore, we benchmark results with state-of-the-art methods for various multi-modal ISLR settings on MM-WLAuslan, including multi-view, cross-camera, and cross-view. Experiment results indicate that MM-WLAuslan is a challenging ISLR dataset, and we hope this dataset will contribute to the development of Auslan and the advancement of sign languages worldwide. All datasets and benchmarks are available at MM-WLAuslan.

Paper Structure

This paper contains 15 sections, 1 equation, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Isolated Sign Language Recognition Datasets
  4. Isolated Sign Language Recognition Methods
  5. Multi-view and Multi-modal Action Recognition
  6. Proposed MM-WLAuslan Dataset
  7. Recording Setup and Workflow
  8. Data Processing and Augmentation
  9. Data Statistics
  10. MM-WLAuslan Benchmark
  11. Isolated Sign Language Recognition Settings
  12. Evaluation Metric
  13. Benchmark Results
  14. Limitation and Future Work
  15. Conclusion

Figures (4)

  • Figure 1: Illustrations of the curated MM-WLAuslan dataset. MM-WLAuslan includes three Kinect-V2 cameras and a RealSense camera arranged hemispherically around the front half of the signer to capture multi-view and multi-modal data.
  • Figure 2: Demonstrations of test subsets. "STU", "ITW", "SYN", and "TED" represent the studio set, in-the-wild set, synthetic background set and temporal disturbance set, respectively.
  • Figure 2: Key statistics of MM-WLAuslan dataset splits. "BG" and "TP" represent background and temporal, respectively. "OOS" indicates the signers only occur in the test set.
  • Figure 3: Statistics of signers and glosses. (a) Ethnicity and gender distribution. (b) Frequency of polysemous glosses. (c) Distribution of Auslan proficiency. (d) Categories of words.