SIG-Chat: Spatial Intent-Guided Conversational Gesture Generation Involving How, When and Where

TL;DR

This work tackles the limitation of existing co-speech gesture systems that lack explicit spatial grounding by introducing SIG-Chat, a multimodal dataset annotated with spatial intents and 3D target positions. It proposes a diffusion-transformer baseline that jointly processes audio, transcripts, and spatial intent through dedicated multimodal encoders and two fusion modules, enabling WHEN and WHERE to guide gestures alongside HOW. The authors define new evaluation metrics (IAR@k and IoU@k) to quantify temporal and angular accuracy, and demonstrate strong improvements over baselines, supported by ablations and a user study. The method is deployed on a humanoid robot, showcasing practical applicability in robotics, games, and animation where interactive, context-aware gestures are essential.

Abstract

The accompanying actions and gestures in dialogue are often closely linked to interactions with the environment, such as looking toward the interlocutor or using gestures to point to the described target at appropriate moments. Speech and semantics guide the production of gestures by determining their timing (WHEN) and style (HOW), while the spatial locations of interactive objects dictate their directional execution (WHERE). Existing approaches either rely solely on descriptive language to generate motions or utilize audio to produce non-interactive gestures, thereby lacking the characterization of interactive timing and spatial intent. This significantly limits the applicability of conversational gesture generation, whether in robotics or in the fields of game and animation production. To address this gap, we present a full-stack solution. We first established a unique data collection method to simultaneously capture high-precision human motion and spatial intent. We then developed a generation model driven by audio, language, and spatial data, alongside dedicated metrics for evaluating interaction timing and spatial accuracy. Finally, we deployed the solution on a humanoid robot, enabling rich, context-aware physical interactions.

Figures (5)

• Figure 1: Driven by audio and spatial intent information, interactive conversational gestures can be generated. Audio determines HOW non-interactive gestures are produced and also WHEN the intent-driven interactions are conducted, such as when referring to an object with phrases like "It's over there." Meanwhile, spatial intent guides WHERE to interact, including the torso and head orienting toward the referent, while the hand pointing toward the target. When deployed on a humanoid robot, this framework allows for conversational gestures capable of interacting with the environment.
• Figure 2: Capture System and Distribution of SIG-Chat. (a) Synchronized Gesture-Inten Recording via MVN motion capture system with HTC Vive. (b) SIG-Chat incorporates 4 initial states. (c) Speakers’ gestures are classified into 3 intent categories. (d) Type distribution of static target positions. (e) Type distribution of dynamic target trajectories.
• Figure 3: Architecture of the denoising network. The model is a multi-layer diffusion-transformer with two fusion modules (speech fusion and intent fusion) that integrates audio, transcript, initial posture description, intent category, and target trajectory as multimodal contitions to estimate the clean gesture $\hat{x}_0$ from noisy gesture $x_t$ at diffusion timestep $t$. Multimodal inputs are processed and encoded by specific encoders. FC refers to the Fully Connected (FC) layer.
• Figure 4: Visualization results of intent-aware gesture generated by SIG-Chat.
• Figure 5: Angular deviation calculation diagrams (a–d) show Index Finger, Hand, Arm, and Gaze Orientation Deviations, with red spheres for intent targets, yellow for key joints, blue arrows for gesture intent directions, and red arrows for expected target direction. Histograms (e) presents statistics of gazing/pointing angular deviations in the SIG-Chat dataset.