Snapture -- A Novel Neural Architecture for Combined Static and Dynamic Hand Gesture Recognition

TL;DR

Snapture tackles the challenge of recognizing both static hand poses and dynamic gestures by fusing a dynamic CNNLSTM channel with a static peak snapshot channel. Motion-profile analysis based on $ISSIM$ and $SSIM$ guides a thresholded static-channel gate (Snapturethold) to mitigate blur and preserve discriminative hand details, yielding improved per-class performance on GRIT and Montalbano compared with a CNNLSTM baseline. The approach remains RGB-only and modular, enabling future integration of body pose, facial cues, or speech, which can enhance HRI scenarios. Overall, Snapture demonstrates that incorporating hand pose at the peak alongside movement substantially reduces confusion among gestures sharing similar trajectories, advancing practical gesture recognition for robot interaction.

Abstract

As robots are expected to get more involved in people's everyday lives, frameworks that enable intuitive user interfaces are in demand. Hand gesture recognition systems provide a natural way of communication and, thus, are an integral part of seamless Human-Robot Interaction (HRI). Recent years have witnessed an immense evolution of computational models powered by deep learning. However, state-of-the-art models fall short in expanding across different gesture domains, such as emblems and co-speech. In this paper, we propose a novel hybrid hand gesture recognition system. Our architecture enables learning both static and dynamic gestures: by capturing a so-called "snapshot" of the gesture performance at its peak, we integrate the hand pose along with the dynamic movement. Moreover, we present a method for analyzing the motion profile of a gesture to uncover its dynamic characteristics and which allows regulating a static channel based on the amount of motion. Our evaluation demonstrates the superiority of our approach on two gesture benchmarks compared to a CNNLSTM baseline. We also provide an analysis on a gesture class basis that unveils the potential of our Snapture architecture for performance improvements. Thanks to its modular implementation, our framework allows the integration of other multimodal data like facial expressions and head tracking, which are important cues in HRI scenarios, into one architecture. Thus, our work contributes both to gesture recognition research and machine learning applications for non-verbal communication with robots.

Figures (20)

• Figure 1: The motion profile of GRIT gestures "stop", "turn left" and "turn". "stop", "turn left" are paused at their peak, while "turn" is with a repeating pattern due to the continuous intensity across its time span.
• Figure 2: The motion profile of Montalbano gestures "vattene", "vieniqui" and "ok". These co-speech gestures follow Kendon's model kendon2011, hence, they start and end with low intensity and have a clear peak around the midpoint of the timeline.
• Figure 3: An overview of the Snapture framework. The architecture consists of a dynamic and static channels, fused into a final classifier. Thus, it performs a hybrid hand gesture recognition task.
• Figure 4: The dynamic channel of Snapture is a CNNLSTM network, consisting of two layers of CNN followed by a LSTM and feed forward network. The isolated gesture input is pre-segmented using the differential image algorithm. For clarity, we show only five frames and increase the contrast of the differential images.
• Figure 5: The five gesture phases of Kendon kendon2011. Each gesture starts with a rest phase. In pre-stroke, the limb moves from the rest position into the stroke phase. The stroke phase contains the most expressive information. In post-stroke, the limb moves away from stroke back into rest phase.