Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GestureGPT: Toward Zero-Shot Free-Form Hand Gesture Understanding with Large Language Model Agents

Xin Zeng, Xiaoyu Wang, Tengxiang Zhang, Chun Yu, Shengdong Zhao, Yiqiang Chen

TL;DR

G gestureGPT is introduced, a free-form hand gesture understanding framework that mimics human gesture understanding procedures to enable a natural free-form gestural interface and validated offline under two real-world scenarios: smart home control and online video streaming.

Abstract

Existing gesture interfaces only work with a fixed set of gestures defined either by interface designers or by users themselves, which introduces learning or demonstration efforts that diminish their naturalness. Humans, on the other hand, understand free-form gestures by synthesizing the gesture, context, experience, and common sense. In this way, the user does not need to learn, demonstrate, or associate gestures. We introduce GestureGPT, a free-form hand gesture understanding framework that mimics human gesture understanding procedures to enable a natural free-form gestural interface. Our framework leverages multiple Large Language Model agents to manage and synthesize gesture and context information, then infers the interaction intent by associating the gesture with an interface function. More specifically, our triple-agent framework includes a Gesture Description Agent that automatically segments and formulates natural language descriptions of hand poses and movements based on hand landmark coordinates. The description is deciphered by a Gesture Inference Agent through self-reasoning and querying about the interaction context (e.g., interaction history, gaze data), which is managed by a Context Management Agent. Following iterative exchanges, the Gesture Inference Agent discerns the user's intent by grounding it to an interactive function. We validated our framework offline under two real-world scenarios: smart home control and online video streaming. The average zero-shot Top-1/Top-5 grounding accuracies are 44.79%/83.59% for smart home tasks and 37.50%/73.44% for video streaming tasks. We also provide an extensive discussion that includes rationale for model selection, generalizability, and future research directions for a practical system etc.

GestureGPT: Toward Zero-Shot Free-Form Hand Gesture Understanding with Large Language Model Agents

TL;DR

G gestureGPT is introduced, a free-form hand gesture understanding framework that mimics human gesture understanding procedures to enable a natural free-form gestural interface and validated offline under two real-world scenarios: smart home control and online video streaming.

Abstract

Existing gesture interfaces only work with a fixed set of gestures defined either by interface designers or by users themselves, which introduces learning or demonstration efforts that diminish their naturalness. Humans, on the other hand, understand free-form gestures by synthesizing the gesture, context, experience, and common sense. In this way, the user does not need to learn, demonstrate, or associate gestures. We introduce GestureGPT, a free-form hand gesture understanding framework that mimics human gesture understanding procedures to enable a natural free-form gestural interface. Our framework leverages multiple Large Language Model agents to manage and synthesize gesture and context information, then infers the interaction intent by associating the gesture with an interface function. More specifically, our triple-agent framework includes a Gesture Description Agent that automatically segments and formulates natural language descriptions of hand poses and movements based on hand landmark coordinates. The description is deciphered by a Gesture Inference Agent through self-reasoning and querying about the interaction context (e.g., interaction history, gaze data), which is managed by a Context Management Agent. Following iterative exchanges, the Gesture Inference Agent discerns the user's intent by grounding it to an interactive function. We validated our framework offline under two real-world scenarios: smart home control and online video streaming. The average zero-shot Top-1/Top-5 grounding accuracies are 44.79%/83.59% for smart home tasks and 37.50%/73.44% for video streaming tasks. We also provide an extensive discussion that includes rationale for model selection, generalizability, and future research directions for a practical system etc.
Paper Structure (46 sections, 11 figures, 9 tables, 5 algorithms)

This paper contains 46 sections, 11 figures, 9 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. LLM as Autonomous Agent
  4. Natural Free-form Gesture Understanding
  5. Gesture Understanding with Context Information
  6. Method
  7. Gesture Description Agent
  8. Rule-Based Gesture Description Generation
  9. Gesture Summary Description Generation
  10. Gesture Inference Agent
  11. Context Management Agent
  12. Context Library Operations
  13. Evaluation
  14. Experiment 1: Augmented Reality-Based Smart Home IoT Control
  15. Experiment Setting and Procedure
  16. ...and 31 more sections

Figures (11)

  • Figure 1: Overview of current gestural interaction systems.
  • Figure 2: Agents Collaboration Workflow.
  • Figure 3: Illustration of gesture description rules. (a) The flexion of a finger is calculated from the sum of bending angle of ip joint (for thumb) or pip and dip joint (for other fingers). (b) The proximity of two fingers is calculated from the average distance from each finger's pip/dip/tip joint to the other finger. (c) The contact of thumb and another finger is calculated from the distance of their fingertips. (d) The pointing direction of thumb is calculated from the vector from thumb's mcp to tip. (e) The palm orientation is calculated from the dot product of the two vectors on the hand, pointing towards the reader.
  • Figure 4: Prompt for Gesture Description Agent.
  • Figure 5: Prompt for Gesture Inference Agent.
  • ...and 6 more figures