Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Logos as a Well-Tempered Pre-train for Sign Language Recognition

Ilya Ovodov, Petr Surovtsev, Karina Kvanchiani, Alexander Kapitanov, Alexander Nagaev

TL;DR

This work tackles cross-language isolated sign language recognition under data scarcity and semantic ambiguity by introducing Logos, a large-scale Russian Sign Language dataset with explicit grouping of visually similar signs (VSSigns). It demonstrates that a model pre-trained on Logos can serve as a universal encoder for other sign languages and that cross-language multi-dataset co-training with language-specific heads yields strong performance, achieving state-of-the-art results on WLASL with RGB input and competitive results on AUTSL. The study also shows that explicitly grouping VSSigns improves downstream transfer, and that dataset size and labeling strategy are critical for cross-language SL transfer. Overall, the Logos approach offers a practical, scalable path toward universal SLR encoders and improved cross-language transfer, with the dataset and code publicly available for research use.

Abstract

This paper examines two aspects of the isolated sign language recognition (ISLR) task. First, although a certain number of datasets is available, the data for individual sign languages is limited. It poses the challenge of cross-language ISLR model training, including transfer learning. Second, similar signs can have different semantic meanings. It leads to ambiguity in dataset labeling and raises the question of the best policy for annotating such signs. To address these issues, this study presents Logos, a novel Russian Sign Language (RSL) dataset, the most extensive available ISLR dataset by the number of signers, one of the most extensive datasets in size and vocabulary, and the largest RSL dataset. It is shown that a model, pre-trained on the Logos dataset can be used as a universal encoder for other language SLR tasks, including few-shot learning. We explore cross-language transfer learning approaches and find that joint training using multiple classification heads benefits accuracy for the target low-resource datasets the most. The key feature of the Logos dataset is explicitly annotated visually similar sign groups. We show that explicitly labeling visually similar signs improves trained model quality as a visual encoder for downstream tasks. Based on the proposed contributions, we outperform current state-of-the-art results for the WLASL dataset and get competitive results for the AUTSL dataset, with a single stream model processing solely RGB video. The source code, dataset, and pre-trained models are publicly available.

Logos as a Well-Tempered Pre-train for Sign Language Recognition

TL;DR

This work tackles cross-language isolated sign language recognition under data scarcity and semantic ambiguity by introducing Logos, a large-scale Russian Sign Language dataset with explicit grouping of visually similar signs (VSSigns). It demonstrates that a model pre-trained on Logos can serve as a universal encoder for other sign languages and that cross-language multi-dataset co-training with language-specific heads yields strong performance, achieving state-of-the-art results on WLASL with RGB input and competitive results on AUTSL. The study also shows that explicitly grouping VSSigns improves downstream transfer, and that dataset size and labeling strategy are critical for cross-language SL transfer. Overall, the Logos approach offers a practical, scalable path toward universal SLR encoders and improved cross-language transfer, with the dataset and code publicly available for research use.

Abstract

This paper examines two aspects of the isolated sign language recognition (ISLR) task. First, although a certain number of datasets is available, the data for individual sign languages is limited. It poses the challenge of cross-language ISLR model training, including transfer learning. Second, similar signs can have different semantic meanings. It leads to ambiguity in dataset labeling and raises the question of the best policy for annotating such signs. To address these issues, this study presents Logos, a novel Russian Sign Language (RSL) dataset, the most extensive available ISLR dataset by the number of signers, one of the most extensive datasets in size and vocabulary, and the largest RSL dataset. It is shown that a model, pre-trained on the Logos dataset can be used as a universal encoder for other language SLR tasks, including few-shot learning. We explore cross-language transfer learning approaches and find that joint training using multiple classification heads benefits accuracy for the target low-resource datasets the most. The key feature of the Logos dataset is explicitly annotated visually similar sign groups. We show that explicitly labeling visually similar signs improves trained model quality as a visual encoder for downstream tasks. Based on the proposed contributions, we outperform current state-of-the-art results for the WLASL dataset and get competitive results for the AUTSL dataset, with a single stream model processing solely RGB video. The source code, dataset, and pre-trained models are publicly available.
Paper Structure (34 sections, 3 figures, 8 tables, 1 algorithm)

This paper contains 34 sections, 3 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Isolated Sign Language Recognition
  4. ISLR Datasets
  5. The VSSigns Problem
  6. Multi-Dataset Training
  7. Logos Dataset
  8. Dataset Characteristics
  9. Gloss Selection
  10. Data Collection
  11. VSSigns Grouping
  12. Train-test Split
  13. Experiments setup
  14. Datasets
  15. Sign Language Recognition Pipeline
  16. ...and 19 more sections

Figures (3)

  • Figure 1: Sample frames from Russian Sign Language dataset Logos: (a,b) and (c,d) are visually similar signs (VSSigns).
  • Figure 2: Multi-dataset co-training pipeline. Samples from different languages are processed as a united batch. Before the inter-sample augmentations and the language-specific classification heads, the language-specific gates split the batch into language-specific sub-batches.
  • Figure 3: Dataset characteristics and distribution analysis. a) Sign length distribution. b) Distance distribution. The distance (in meters) is approximately estimated based on the length between the left and right shoulders of the signer obtained using MediaPipe mediapipe. c) Signers' devices. d) Devices resolution. e) Number of videos per signer. f) Brightness distribution. The sample brightness is the mean pixel brightness of grayscaled video frames. g) Signers' gender; h) Signers' age. The age is determined by the MiVOLO model kuprashevich2023mivolo.