MASA: Motion-aware Masked Autoencoder with Semantic Alignment for Sign Language Recognition

TL;DR

This work addresses the insufficiency of motion modeling and global semantic guidance in self-supervised pre-training for isolated sign language recognition. It introduces MASA, a two-component framework comprising a Motion-aware Masked Autoencoder (MA) and a Momentum Semantic Alignment (SA) module, to jointly capture local motion cues and global lexical semantics. The pre-training objective combines a motion residual prediction loss and a contrastive semantic consistency loss, enabling robust representations that improve performance across four benchmarks. MASA demonstrates state-of-the-art results and highlights the value of integrating motion-aware masking with global semantic alignment for sign language understanding, with potential extensions to multi-scale motion and RGB data.

Abstract

Sign language recognition (SLR) has long been plagued by insufficient model representation capabilities. Although current pre-training approaches have alleviated this dilemma to some extent and yielded promising performance by employing various pretext tasks on sign pose data, these methods still suffer from two primary limitations: 1) Explicit motion information is usually disregarded in previous pretext tasks, leading to partial information loss and limited representation capability. 2) Previous methods focus on the local context of a sign pose sequence, without incorporating the guidance of the global meaning of lexical signs. To this end, we propose a Motion-Aware masked autoencoder with Semantic Alignment (MASA) that integrates rich motion cues and global semantic information in a self-supervised learning paradigm for SLR. Our framework contains two crucial components, i.e., a motion-aware masked autoencoder (MA) and a momentum semantic alignment module (SA). Specifically, in MA, we introduce an autoencoder architecture with a motion-aware masked strategy to reconstruct motion residuals of masked frames, thereby explicitly exploring dynamic motion cues among sign pose sequences. Moreover, in SA, we embed our framework with global semantic awareness by aligning the embeddings of different augmented samples from the input sequence in the shared latent space. In this way, our framework can simultaneously learn local motion cues and global semantic features for comprehensive sign language representation. Furthermore, we conduct extensive experiments to validate the effectiveness of our method, achieving new state-of-the-art performance on four public benchmarks.

Figures (2)

• Figure 1: The illustration of our proposed framework during pre-training. It mainly contains two crucial components, i.e., a motion-aware masked autoencoder (MA) and a momentum semantic alignment module (SA). In MA, given a sign pose sequence, the frame-wise embedding layer encodes each frame pose into a latent feature. The encoder operates on unmasked feature sequences. Then, the target decoder predicts the motion residuals from pose tokens (along with mask tokens). In SA, it employs a siamese framework extracting pose features from randomly sampled pose sequence similar to MA. After that, we project global features from positive views into a shared embedding space and align them with the semantic consistency loss. The modules with the black dashed box are momentum updated and the red arrows represent the gradient back propagation.
• Figure 2: Effect of the data scale during pre-training on WLASL dataset. The horizontal coordinate denotes the data ratio, while the vertical coordinate denotes recognition accuracy.