Recognition of Dynamic Hand Gestures in Long Distance using a Web-Camera for Robot Guidance

TL;DR

This work tackles long-range dynamic gesture recognition for robot guidance using a web-camera, addressing recognition from distances up to $20$ m. It proposes SlowFast-Transformer (SFT), integrating multi-timecale video features with a Transformer for temporal modeling, and a distance-aware loss, LongLoss, to improve robustness to user distance. The approach outperforms contemporary video-recognition models, achieving up to $95.7\%$ accuracy on a diverse dataset of $N=3{,}240$ samples collected from 4–20 m, and demonstrates stability across distance. This has practical implications for intuitive, hands-free robot control in real-world settings, with potential for vocabulary expansion and real-time optimization in future work.

Abstract

Dynamic gestures enable the transfer of directive information to a robot. Moreover, the ability of a robot to recognize them from a long distance makes communication more effective and practical. However, current state-of-the-art models for dynamic gestures exhibit limitations in recognition distance, typically achieving effective performance only within a few meters. In this work, we propose a model for recognizing dynamic gestures from a long distance of up to 20 meters. The model integrates the SlowFast and Transformer architectures (SFT) to effectively process and classify complex gesture sequences captured in video frames. SFT demonstrates superior performance over existing models.

Figures (3)

• Figure 1: Directing a robot through obstacles using gestures.
• Figure 2: Nine hand gestures used in this study.
• Figure 3: Gesture recognition success rate of the SFT model with regard to the distance $d$ of the user from the camera.