HandReader: Advanced Techniques for Efficient Fingerspelling Recognition

TL;DR

The paper tackles fingerspelling recognition in sign language under variable video lengths and real-world conditions. It introduces HandReader, a trio of architectures that leverage RGB (HandReader_RGB), keypoints (HandReader_KP), and a joint RGB+KP (HandReader_RGB+KP) encoder, all trained with CTC and a GRU decoder, and powered by two novel modules: TSAM for dynamic RGB processing and TPE for pose-based keypoint encoding. Key contributions include the Temporal Shift-Adaptive Module (TSAM) that handles variable-length sequences without padding or trimming, the Temporal Pose Encoder (TPE) that extracts spatio-temporal information from 54 keypoints, and the Znaki Russian fingerspelling dataset released to the community, along with pre-trained models. Empirical results show state-of-the-art letter accuracy on ChicagoFSWild and ChicagoFSWild+ and strong performance on Znaki, with ablation studies highlighting the importance of augmentations, video-length handling, and modality fusion. The work advances practical fingerspelling recognition and provides valuable resources for Russian SL research, while acknowledging limitations related to auxiliary detectors, inference speed, and dataset biases.

Abstract

Fingerspelling is a significant component of Sign Language (SL), allowing the interpretation of proper names, characterized by fast hand movements during signing. Although previous works on fingerspelling recognition have focused on processing the temporal dimension of videos, there remains room for improving the accuracy of these approaches. This paper introduces HandReader, a group of three architectures designed to address the fingerspelling recognition task. HandReader$_{RGB}$ employs the novel Temporal Shift-Adaptive Module (TSAM) to process RGB features from videos of varying lengths while preserving important sequential information. HandReader$_{KP}$ is built on the proposed Temporal Pose Encoder (TPE) operated on keypoints as tensors. Such keypoints composition in a batch allows the encoder to pass them through 2D and 3D convolution layers, utilizing temporal and spatial information and accumulating keypoints coordinates. We also introduce HandReader_RGB+KP - architecture with a joint encoder to benefit from RGB and keypoint modalities. Each HandReader model possesses distinct advantages and achieves state-of-the-art results on the ChicagoFSWild and ChicagoFSWild+ datasets. Moreover, the models demonstrate high performance on the first open dataset for Russian fingerspelling, Znaki, presented in this paper. The Znaki dataset and HandReader pre-trained models are publicly available.

Figures (5)

• Figure 1: Three HandReader architectures. a) The videos with different lengths b) are processed to get crops and keypoints for HandReader$_{RGB}$ and HandReader$_{KP}$, respectively, c) form the batches, and d) pass through the encoders. Further, e) the GRU decoder is applied to the encoder outputs. The final output of the HandReader$_{RGB+KP}$ is obtained by summing the outputs of TSAM and TPE. Red-colored encoders are the paper's contribution.
• Figure 2: Comparison of the TSAM architecture (outlined in red at the top) with the original TSM (outlined in gray at the bottom). The input batch includes two videos of different lengths, with specific frames highlighted in blue and yellow. While the TSM model requires fixed-length inputs utilizing padding (shown in white) or trimming frames (shown in red) when necessary, the TSAM model employs individual shift counters to manage variable-length sequences without additional padding or trimming. The shift counter displayed at the top of the TSAM block shows how the shifts are activated individually for each video, incrementing the shift counter until it reaches the length of that specific video.
• Figure 3: Demonstration of the Temporal Pose Encoder operating on a single batch. a) The encoder receives an input tensor containing $N$ frames and $K$ keypoints. All videos are padded to match the length of the most extended video clip in the batch, allowing them to be stacked. b) 2D convolutions process multiple keypoints across several frames, extracting temporal features. c) Subsequently, a 3D convolution in the form of a "tube" aggregates the extracted motion information for each body part into final feature representations $F$. d) After applying TPE, features are processed with linear layer, e) followed by MLP with convolution modules from Conformer.
• Figure 4: Znaki dataset. a) video lengths distribution; b) total number of letters distribution; c) distance distribution: the distance is approximately estimated in meters by computed the length between the signers' left and right shoulders using MediaPipe mediapipe; d) letter frequency: comparison of letters frequency in the dataset Znaki and the Russian language corpus.
• Figure 5: Sample frames from the Znaki dataset.