Vision-Based Hand Gesture Customization from a Single Demonstration

TL;DR

This work tackles the problem of enabling users to define and deploy personalized hand gestures from a single demonstration in vision-based HCI. It combines 2D hand keypoint extraction, a graph transformer to capture spatial relations, and model-agnostic meta-learning with meta-augmentation to achieve rapid adaptation to new gestures, including static/dynamic and one-/two-handed variants across egocentric and allocentric viewpoints. Key contributions include an end-to-end customization approach that handles a background (null) class, on-device training, and evaluation on 21 participants across 20 new gestures, achieving up to 95% accuracy with one demonstration, plus three practical applications (design tool, video editing, and MR). The proposed method enables flexible, data-efficient gesture customization suitable for real-time deployment and broad accessibility, paving the way for user-defined gestures in diverse interactive contexts.

Abstract

Hand gesture recognition is becoming a more prevalent mode of human-computer interaction, especially as cameras proliferate across everyday devices. Despite continued progress in this field, gesture customization is often underexplored. Customization is crucial since it enables users to define and demonstrate gestures that are more natural, memorable, and accessible. However, customization requires efficient usage of user-provided data. We introduce a method that enables users to easily design bespoke gestures with a monocular camera from one demonstration. We employ transformers and meta-learning techniques to address few-shot learning challenges. Unlike prior work, our method supports any combination of one-handed, two-handed, static, and dynamic gestures, including different viewpoints, and the ability to handle irrelevant hand movements. We implement three real-world applications using our customization method, conduct a user study, and achieve up to 94% average recognition accuracy from one demonstration. Our work provides a viable path for vision-based gesture customization, laying the foundation for future advancements in this domain.

Figures (10)

• Figure 1: Overview of our method's pipeline: We use a hand pose estimation model (A) to extract hand skeleton and landmarks. We utilize a graph transformer to capture both (B) joint-to-joint and (C) joint-to-group attentions. Temporal features are extracted using (D) temporal convolutions. We implement and enhance customization and few-shot learning (FSL) through (E) meta-augmentation and (F) meta-learning approaches.
• Figure 2: Graph transformer architecture used in our method. The graph transformer takes the keypoints and joint groups (fingers), projects them into higher dimensions and calculates attention maps to extract spatial features. The spatial and temporal features are then aggregated through temporal convolutions to yield the final output.
• Figure 3: Visualization of the transformer's joint-to-joint attention maps for eight gestures. Each map is either a $21\times21$ (one-handed) or $42\times42$ (two-handed) matrix averaged across all attention heads ($N=9$). The model attends to different parts of the hand based on the gesture. For example, for a swipe right gesture, most keypoints attend to the thumb and ring finger.
• Figure 4: Shown here are the 20 gestures on which we evaluated our model. The blue square indicates the dynamic one-handed gestures, while the yellow diamond indicates the static one-handed gestures. Our dataset also comprises of six two-handed gestures, indicated by the green triangle for dynamic gestures and the red circle for static gestures. Our method can accommodate most gestures that a user might present from these gesture categories. This particular evaluation set was selected to represent the breadth of possible gestures, but should not be taken to suggest that our system can only accommodate these gestures.
• Figure 5: Accuracy of our method and baselines when tested with (A) two, (B) three, and (C) four new gestures along with the background class. In (D) we show a specific experiment where our method achieves its maximum accuracy improvement over baseline models such as the random forest. In (E) we show a specific experiment where our method achieves its minimum accuracy improvement over baseline models such as the random forest. For a complete evaluation and details, see Appendix \ref{['appendix:A']}.