Gestura: A LVLM-Powered System Bridging Motion and Semantics for Real-Time Free-Form Gesture Understanding

TL;DR

Gestura introduces a LVLM-powered system for real-time free-form gesture understanding by embedding anatomical hand priors through a Landmark Processing Module and guiding cross-modal reasoning with Chain-of-Thought tuning. It also presents GestureInt, the first open-source dataset for gesture meaning and intention in a QA framework, enabling robust open-world evaluation. The method uses a two-stage training regime—semantic alignment with a frozen visual/Text backbone, followed by CoT-enhanced cross-modal learning with landmark signals—yielding strong exocentric and egocentric performance and real-time inference suitable for edge devices. Empirical results show Gestura outperforms prior LVLM approaches by large margins on both intent recognition and gesture description, with practical end-to-end latency and the potential for deployment in wearable AR/VR and smart-home contexts.

Abstract

Free-form gesture understanding is highly appealing for human-computer interaction, as it liberates users from the constraints of predefined gesture categories. However, the sole existing solution GestureGPT suffers from limited recognition accuracy and slow response times. In this paper, we propose Gestura, an end-to-end system for free-form gesture understanding. Gestura harnesses a pre-trained Large Vision-Language Model (LVLM) to align the highly dynamic and diverse patterns of free-form gestures with high-level semantic concepts. To better capture subtle hand movements across different styles, we introduce a Landmark Processing Module that compensate for LVLMs' lack of fine-grained domain knowledge by embedding anatomical hand priors. Further, a Chain-of-Thought (CoT) reasoning strategy enables step-by-step semantic inference, transforming shallow knowledge into deep semantic understanding and significantly enhancing the model's ability to interpret ambiguous or unconventional gestures. Together, these components allow Gestura to achieve robust and adaptable free-form gesture comprehension. Additionally, we have developed the first open-source dataset for free-form gesture intention reasoning and understanding with over 300,000 annotated QA pairs.

Figures (7)

• Figure 1: Overview of the proposed framework of Gestura. Gestura introduces a hierarchical framework with two-phase training for free-form gesture understanding. First, the pre-training stage activates the model's potential for free-form generalization by using a multi-view semantic enhancement strategy. In Stage 2, Gestura leverages well-trained landmark processing module to deliver anatomical and spatial contextual signals to the LVLM backbone, while internalizing advanced reasoning capabilities through Chain-of-Thought (CoT) tuning. This dual mechanism expands the model’s reasoning boundaries, enabling superior generalization in free-form gesture understanding.
• Figure 2: The process of deriving encoded features from hand video data. [R1, R4] A. Raw RGB frame → B. 21 MediaPipe landmarks → C. One landmark triplet (red) case from the 1,330 possible → D. Measure the enclosed angle. Blue circles mark the 21 hand landmarks detected by MediaPipe. One triplet of landmarks (red) is chosen out of the $\binom{21}{3}=1{,}330$ possible combinations each time to form two vectors whose enclosed angle $\alpha$ is measured. The cosine of this angle constitutes a single element of the final feature vector.
• Figure 3: Score rating and comparison of answers with and without CoT tuning. Certain portions of the generated output are highlighted to underscore their critical role in the inference process leading to the final answer.
• Figure 4: The annotation pipeline
• Figure 5: Top-k Accuracy with 95% Bootstrap CI