ImaGGen: Zero-Shot Generation of Co-Speech Semantic Gestures Grounded in Language and Image Input

TL;DR

ImaGGen presents a fully automatic, zero-shot system that grounds co-speech gestures in both language and image input to produce iconic, deictic, and beat gestures without human annotation. The architecture fuses an image feature pipeline, a semantic matching module, and a differentiable realization engine with a two-pass IK solver, enabling synchronized, semantically informative gestures. A comprehensive user study shows that iconic and deictic gestures substantially improve interpretation in ambiguous speech contexts and enhance perceived coherence, though complex shapes remain challenging and social appropriateness can modulate user experience. The work advances embodied agents by enabling context-aware multimodal gestures that communicate visual properties from images, paving the way for more natural and collaborative human-agent interactions.

Abstract

Human communication combines speech with expressive nonverbal cues such as hand gestures that serve manifold communicative functions. Yet, current generative gesture generation approaches are restricted to simple, repetitive beat gestures that accompany the rhythm of speaking but do not contribute to communicating semantic meaning. This paper tackles a core challenge in co-speech gesture synthesis: generating iconic or deictic gestures that are semantically coherent with a verbal utterance. Such gestures cannot be derived from language input alone, which inherently lacks the visual meaning that is often carried autonomously by gestures. We therefore introduce a zero-shot system that generates gestures from a given language input and additionally is informed by imagistic input, without manual annotation or human intervention. Our method integrates an image analysis pipeline that extracts key object properties such as shape, symmetry, and alignment, together with a semantic matching module that links these visual details to spoken text. An inverse kinematics engine then synthesizes iconic and deictic gestures and combines them with co-generated natural beat gestures for coherent multimodal communication. A comprehensive user study demonstrates the effectiveness of our approach. In scenarios where speech alone was ambiguous, gestures generated by our system significantly improved participants' ability to identify object properties, confirming their interpretability and communicative value. While challenges remain in representing complex shapes, our results highlight the importance of context-aware semantic gestures for creating expressive and collaborative virtual agents or avatars, marking a substantial step forward towards efficient and robust, embodied human-agent interaction. More information and example videos are available here: https://review-anon-io.github.io/ImaGGen.github.io/

Figures (8)

• Figure 1: Architecture of the ImaGGen model with its three main components: Image Feature Analysis, Semantic Matching Pipeline, and Realization Engine
• Figure 2: Illustration of all Image Feature Analysis steps on an example image. From left to right: Bounding Box detection, Segmentation, Object Symmetry Analysis, Object Alignment Analysis, Primitive Analysis, Trajectory Analysis.
• Figure 3: Example of the performed user study. First, the video has to be watched before selecting one of the four available images and a confidence value.
• Figure 4: Mean accuracy for selecting the correct image from four options for each condition, grouped by the five tested categories with corresponding standard deviations. Statistical significance is only shown for the 'No Info + Iconic' condition. Higher values indicate better performance. ( * = p $<$ 0.05, ** = p $<$ 0.005, *** = p $<$ 0.0005, **** = p $<$ 0.00005)
• Figure 5: Proportional response rates for selecting the correct image, an incorrect image, or the fallback option "None of these", across all conditions, grouped by tested categories. ( * = p $<$ 0.05, ** = p $<$ 0.005, *** = p $<$ 0.0005, **** = p $<$ 0.00005)