StepNet: Spatial-temporal Part-aware Network for Isolated Sign Language Recognition

TL;DR

Isolated sign language recognition benefits from both fine-grained appearance and temporal context. StepNet introduces two parallel branches—Part-level Spatial Modeling for local hand-face relationships and Part-level Temporal Modeling for long-short term dynamics—operating on RGB features without keypoint annotations and fused at the end. The method employs spatial partitions (left-right, top-bottom, global) with a gating mechanism and attention, and temporal partitions with GRUs plus temporal attention to capture multi-scale cues, supervised by a multi-term cross-entropy loss $L_{total}$. A Two-Stream extension with optical flow further boosts performance, achieving competitive Top-1 accuracies on WLASL, NMFs-CSL, and BOBSL, and demonstrating robustness across sign languages. This RGB-based, part-aware approach advances SLR by better preserving appearance details while modeling temporal evolution, reducing reliance on noisy keypoints and enabling effective multi-cue fusion.

Abstract

The goal of sign language recognition (SLR) is to help those who are hard of hearing or deaf overcome the communication barrier. Most existing approaches can be typically divided into two lines, i.e., Skeleton-based and RGB-based methods, but both the two lines of methods have their limitations. Skeleton-based methods do not consider facial expressions, while RGB-based approaches usually ignore the fine-grained hand structure. To overcome both limitations, we propose a new framework called Spatial-temporal Part-aware network~(StepNet), based on RGB parts. As its name suggests, it is made up of two modules: Part-level Spatial Modeling and Part-level Temporal Modeling. Part-level Spatial Modeling, in particular, automatically captures the appearance-based properties, such as hands and faces, in the feature space without the use of any keypoint-level annotations. On the other hand, Part-level Temporal Modeling implicitly mines the long-short term context to capture the relevant attributes over time. Extensive experiments demonstrate that our StepNet, thanks to spatial-temporal modules, achieves competitive Top-1 Per-instance accuracy on three commonly-used SLR benchmarks, i.e., 56.89% on WLASL, 77.2% on NMFs-CSL, and 77.1% on BOBSL. Additionally, the proposed method is compatible with the optical flow input and can produce superior performance if fused. For those who are hard of hearing, we hope that our work can act as a preliminary step.

Figures (6)

• Figure 1: Selected challenging examples from the widely-used sign language WLASL benchmark li2020word. We could observe that there are some similar-appearance hand gestures, e.g., "wish" (top) and "hungry" (bottom). Such cases demand the learned model to mine more fine-grained details, such as facial expressions.
• Figure 2: Our motivation for designing the part-aware modules. (a) Part-level Spatial Partition: We can infer what the sign video means by looking at just one part, either the green part (shaking head) or the brown part (closing three fingers) (b) Part-level Temporal Partition: Similarly, we observe that any three short video clips, i.e., red, yellow, and blue, can represent the gloss of the sign video. These two kinds of partition inspire us to harness RGB parts to mine fine-grained features for sign language recognition.
• Figure 3: (a) A schematic of our framework. StepNet consists of four parts, i.e., backbone, Part-level Spatial Modeling, Part-level Temporal Modeling, and classification layer. The backbone is utilized to extract feature maps of inputs $\textbf{M}$. The feature maps $\textbf{M}$ are then processed in parallel using a Part-level Spatial Modeling and a Part-level Temporal Modeling. Finally, outputs of two part-level modeling are fused and put into the classification layer to obtain classification logit vectors $q^{rgb}_{st}$. Sub-figure (b),(c). Details of the spatial partition and attention. Spatial partition includes the local guidance, i.e., left-right(lr) and top-bottom (tb), and the global guidance (sg). The lr and tb are composed of Pooling (P) and Gate (G) operators, which build the channel relationships between the lr or tb. Global guidance(sg) guides the model to obtain a coarse but global representation $g_{sg}$. After that, we introduce spatial attention in Sub-figure (c) to compute how to aggregate these features $g_{lr}, g_{tb}, g_{sg}$. Sub-figure (d),(e). Details of the temporal partition and attention. We first pool the feature maps and then split them along the temporal dimension into three segments, i.e., $s_1, s_2, s_3$. Each segment is utilized to explore the short clip context by $GRU$. In addition, we also apply the $GRU$ to the long clip (before partition), which models the long-term representation $g_{tg}$. After that, temporal attention learns how to complement the long-term representation $g_{tg}$ from the short-term context. Sub-figure (f). An example of partition with overlaps along the temporal axis.
• Figure 4: A schematic of our two-stream fusion method. We fuse the classification logits ($q^{rgb}_{st}, q^{opt}_{st}$) processed from two StepNets to obtain the final logits ($q_{r+o}$). Notably, two StepNets do not share weights.
• Figure 5: The Per-instance accuracy of each classifier on WLASL-2000 li2020word.