Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Modeling Turn-Taking with Semantically Informed Gestures

Varsha Suresh, M. Hamza Mughal, Christian Theobalt, Vera Demberg

TL;DR

This paper tackles turn-taking prediction in multiparty conversations by leveraging semantically annotated gestures. It introduces DnD Gesture++, a richly labeled extension of the DnD Gesture corpus with 2,663 gesture-type annotations across iconic, metaphoric, deictic, and discourse categories, reformatted into 12k turn events. A Mixture-of-Experts model fuses text, audio, and semantically informed gesture representations derived from a VQ-VAE, with a gating network balancing modalities. Results show that semantically supervised gestures yield consistent gains in turn-taking prediction, highlighting their complementary role and offering a valuable resource for broader multimodal dialogue tasks.

Abstract

In conversation, humans use multimodal cues, such as speech, gestures, and gaze, to manage turn-taking. While linguistic and acoustic features are informative, gestures provide complementary cues for modeling these transitions. To study this, we introduce DnD Gesture++, an extension of the multi-party DnD Gesture corpus enriched with 2,663 semantic gesture annotations spanning iconic, metaphoric, deictic, and discourse types. Using this dataset, we model turn-taking prediction through a Mixture-of-Experts framework integrating text, audio, and gestures. Experiments show that incorporating semantically guided gestures yields consistent performance gains over baselines, demonstrating their complementary role in multimodal turn-taking.

Modeling Turn-Taking with Semantically Informed Gestures

TL;DR

This paper tackles turn-taking prediction in multiparty conversations by leveraging semantically annotated gestures. It introduces DnD Gesture++, a richly labeled extension of the DnD Gesture corpus with 2,663 gesture-type annotations across iconic, metaphoric, deictic, and discourse categories, reformatted into 12k turn events. A Mixture-of-Experts model fuses text, audio, and semantically informed gesture representations derived from a VQ-VAE, with a gating network balancing modalities. Results show that semantically supervised gestures yield consistent gains in turn-taking prediction, highlighting their complementary role and offering a valuable resource for broader multimodal dialogue tasks.

Abstract

In conversation, humans use multimodal cues, such as speech, gestures, and gaze, to manage turn-taking. While linguistic and acoustic features are informative, gestures provide complementary cues for modeling these transitions. To study this, we introduce DnD Gesture++, an extension of the multi-party DnD Gesture corpus enriched with 2,663 semantic gesture annotations spanning iconic, metaphoric, deictic, and discourse types. Using this dataset, we model turn-taking prediction through a Mixture-of-Experts framework integrating text, audio, and gestures. Experiments show that incorporating semantically guided gestures yields consistent performance gains over baselines, demonstrating their complementary role in multimodal turn-taking.
Paper Structure (16 sections, 5 figures, 3 tables)

This paper contains 16 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Modeling Turn-Taking for Spoken Dialog
  4. Role of Semantic Gestures in Spoken Dialog
  5. DnD Group Gesture++
  6. Gesture type annotation
  7. Converting to turn-taking data
  8. Multimodal Modeling of Turn-Taking Prediction
  9. Experiments
  10. Model Comparisons
  11. Turn-taking Prediction
  12. Analysing gesture representations
  13. Conclusion
  14. Appendix
  15. Additional Annotation Details
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Gesture Type Annotations. Time-aligned labels contain semantic gesture types, that are determined by speech context. These labels can help learn gesture representations which improve turn-taking prediction in a multi-party conversation.
  • Figure 2: MoE modeling of turn taking
  • Figure 3: Learning semantically-aligned gesture representations.
  • Figure 4: Visualization of Gesture Representations.
  • Figure 5: MoE modality weight contributions for semantic vs w/o semantic gesture representations