GesGPT: Speech Gesture Synthesis With Text Parsing from ChatGPT

TL;DR

GesGPT addresses the gap in semantic expressiveness for co-speech gesture synthesis by using ChatGPT-based text parsing to extract intention, emphasis, and semantic cues from input text. The method deploys a three-module pipeline: a text-parsing module that outputs a script, a semantically annotated gesture lexicon, and a gesture integration module that fuses professional gestures with base rhythmic motions learned via a 1D U-Net, producing controllable and expressive gestures. Experiments on English BEAT and Chinese ZHUBO datasets show superior rhythm alignment and semantic expressiveness compared to baselines, validating the approach's effectiveness across languages. This work highlights the potential of large language models in embodied gesture generation and provides a practical, editable framework for semantically grounded co-speech gestures with cross-lingual applicability.

Abstract

Gesture synthesis has gained significant attention as a critical research field, aiming to produce contextually appropriate and natural gestures corresponding to speech or textual input. Although deep learning-based approaches have achieved remarkable progress, they often overlook the rich semantic information present in the text, leading to less expressive and meaningful gestures. In this letter, we propose GesGPT, a novel approach to gesture generation that leverages the semantic analysis capabilities of large language models , such as ChatGPT. By capitalizing on the strengths of LLMs for text analysis, we adopt a controlled approach to generate and integrate professional gestures and base gestures through a text parsing script, resulting in diverse and meaningful gestures. Firstly, our approach involves the development of prompt principles that transform gesture generation into an intention classification problem using ChatGPT. We also conduct further analysis on emphasis words and semantic words to aid in gesture generation. Subsequently, we construct a specialized gesture lexicon with multiple semantic annotations, decoupling the synthesis of gestures into professional gestures and base gestures. Finally, we merge the professional gestures with base gestures. Experimental results demonstrate that GesGPT effectively generates contextually appropriate and expressive gestures.

Figures (6)

• Figure 1: The Pipeline of GesGPT. Firstly, the Text Parsing module is employed to generate a gesture script from the input text. Then, the Gesture Lexicon module is used to search and retrieve corresponding professional gestures based on the script. Next, the Gesture Integration module combines the semantically professional gestures from the gesture lexicon with rhythmic base gestures.
• Figure 2: The Pipeline of Text Parsing. It involves three constituent subtasks: action intent classification, emphasis word recognition, and semantic word recognition. Finally, the results are integrated into a parsing script.
• Figure 3: Illustration of Gesture Clip in Gesture Lexicon. Gestures typically go through complete stages, including rest pose, preparation, stroke, retraction, and return to rest pose. However, not all Gesture clips have all three stages of preparation, stroke, and retraction.
• Figure 4: Professional Gesture Lexicon. The upper graph represents the quantity of gestures belonging to different intent categories. Below is an example of a gesture clip from the lexicon.
• Figure 5: Visualization of Generated Examples on the BEAT Dataset. The first line below each text represents the generated results of GesGPT, and the second line represents the generated results of the baseline.