Conveying Meaning through Gestures: An Investigation into Semantic Co-Speech Gesture Generation

TL;DR

The paper investigates whether explicit semantic enrichment improves co-speech gesture generation by comparing AQ-GT and its semantically augmented variant AQ-GT-a using SAGA-derived sentences and movement-focused contexts. AQ-GT integrates multimodal inputs via a GRU-transformer with a VQ-VAE-2 backbone, while AQ-GT-a adds a semantic input channel from SAGA annotations and an augmented Prediction Network to enhance meaning representation. Results show AQ-GT excels on in-domain, SAGA-like sentences, whereas AQ-GT-a offers better generalization to novel contexts and increases perceived expressiveness without boosting human-likeness. This reveals a context-dependent trade-off: explicit semantic augmentation can improve certain communicative aspects and generalization, but does not universally enhance concept recognition or naturalness, guiding future work on how to integrate semantic cues without constraining data-driven mappings.

Abstract

This study explores two frameworks for co-speech gesture generation, AQ-GT and its semantically-augmented variant AQ-GT-a, to evaluate their ability to convey meaning through gestures and how humans perceive the resulting movements. Using sentences from the SAGA spatial communication corpus, contextually similar sentences, and novel movement-focused sentences, we conducted a user-centered evaluation of concept recognition and human-likeness. Results revealed a nuanced relationship between semantic annotations and performance. The original AQ-GT framework, lacking explicit semantic input, was surprisingly more effective at conveying concepts within its training domain. Conversely, the AQ-GT-a framework demonstrated better generalization, particularly for representing shape and size in novel contexts. While participants rated gestures from AQ-GT-a as more expressive and helpful, they did not perceive them as more human-like. These findings suggest that explicit semantic enrichment does not guarantee improved gesture generation and that its effectiveness is highly dependent on the context, indicating a potential trade-off between specialization and generalization.

Figures (6)

• Figure 1: Example view of stage one of the evaluation study
• Figure 2: Example view of stage two of the evaluation study
• Figure 3: Comparing scores across four evaluation dimensions: Human-Like, Reflection of Speech, Helpful, and Synchronicity for three experimental groups: AQGT, AQGT-A, and AQGT-A Annotated. The y-axis represents the score, ranging from 1.0 to 4.0, with error bars indicating standard deviations.
• Figure 4: Comparison of the scores from the six concepts on sentences derived directly from the SAGA corpus. See \ref{['study_design']} for more information. The asterisks denote the statistical significance levels (p < 0.05, **p < 0.001, ***p < 0.0001)
• Figure 5: Comparison of the scores from the six concepts on sentences similar to those in the saga corpus. See \ref{['study_design']} for more information. The asterisks denote the statistical significance levels (p < 0.05, **p < 0.001, ***p < 0.0001)