Dynamic Spatial-Temporal Aggregation for Skeleton-Aware Sign Language Recognition

TL;DR

A new spatial architecture consisting of two concurrent branches, which build input-sensitive joint relationships and incorporates specific domain knowledge for recognition, respectively are proposed, which achieves state-of-the-art accuracy compared to previous skeleton-aware methods on four large-scale SLR benchmarks.

Abstract

Skeleton-aware sign language recognition (SLR) has gained popularity due to its ability to remain unaffected by background information and its lower computational requirements. Current methods utilize spatial graph modules and temporal modules to capture spatial and temporal features, respectively. However, their spatial graph modules are typically built on fixed graph structures such as graph convolutional networks or a single learnable graph, which only partially explore joint relationships. Additionally, a simple temporal convolution kernel is used to capture temporal information, which may not fully capture the complex movement patterns of different signers. To overcome these limitations, we propose a new spatial architecture consisting of two concurrent branches, which build input-sensitive joint relationships and incorporates specific domain knowledge for recognition, respectively. These two branches are followed by an aggregation process to distinguishe important joint connections. We then propose a new temporal module to model multi-scale temporal information to capture complex human dynamics. Our method achieves state-of-the-art accuracy compared to previous skeleton-aware methods on four large-scale SLR benchmarks. Moreover, our method demonstrates superior accuracy compared to RGB-based methods in most cases while requiring much fewer computational resources, bringing better accuracy-computation trade-off. Code is available at https://github.com/hulianyuyy/DSTA-SLR.

Figures (6)

• Figure 1: Overview for our proposed model.
• Figure 2: Overview for the proposed graph correlation module.
• Figure 3: Overview for the proposed super node transform module.
• Figure 4: Overview for the proposed parallel temporal convolution module.
• Figure 5: Visualizations of learned edges in the graph correlation module for two signs. Only the top 5% active connections are plotted. It's observed that our method learns to build dynamic joint relationships for different input samples, and especially pays attention to the joints of both hands to capture sign movements.