Learning Co-Speech Gesture Representations in Dialogue through Contrastive Learning: An Intrinsic Evaluation

TL;DR

This work tackles learning general-purpose co-speech gesture representations in dialogue by leveraging self-supervised contrastive learning that grounds skeletal gestures in co-occurring speech. It introduces unimodal and multimodal pre-training on a naturalistic referential-dialogue dataset, producing gesture embeddings that correlate with human-annotated form features and reflect dialogue dynamics. The combined unimodal+multimodal objective yields the strongest alignment with expert similarity judgments and enables decoding of gesture properties (e.g., handedness, position) from latent representations, while movement remains challenging. The findings support multimodal contrastive learning as a scalable approach for large-scale gesture analysis and set the stage for integrating richer skeletal and linguistic signals in gesture understanding.

Abstract

In face-to-face dialogues, the form-meaning relationship of co-speech gestures varies depending on contextual factors such as what the gestures refer to and the individual characteristics of speakers. These factors make co-speech gesture representation learning challenging. How can we learn meaningful gestures representations considering gestures' variability and relationship with speech? This paper tackles this challenge by employing self-supervised contrastive learning techniques to learn gesture representations from skeletal and speech information. We propose an approach that includes both unimodal and multimodal pre-training to ground gesture representations in co-occurring speech. For training, we utilize a face-to-face dialogue dataset rich with representational iconic gestures. We conduct thorough intrinsic evaluations of the learned representations through comparison with human-annotated pairwise gesture similarity. Moreover, we perform a diagnostic probing analysis to assess the possibility of recovering interpretable gesture features from the learned representations. Our results show a significant positive correlation with human-annotated gesture similarity and reveal that the similarity between the learned representations is consistent with well-motivated patterns related to the dynamics of dialogue interaction. Moreover, our findings demonstrate that several features concerning the form of gestures can be recovered from the latent representations. Overall, this study shows that multimodal contrastive learning is a promising approach for learning gesture representations, which opens the door to using such representations in larger-scale gesture analysis studies.

Figures (7)

• Figure 1: Pair of gestures referring to the highlighted subpart of the non-conventional object. The gesture pair is coded as similar in all form features: handedness (speakers use both hands), shape, position, rotation, and movement.
• Figure 2: Distribution of the number of shared form features in the gesture pairs manually annotated by Rasenberg et al.
• Figure 3: The proposed contrastive learning framework utilizing both unimodal and multimodal objectives.
• Figure 4: The cosine similarity scores distribution between pairs of gestures' representations, based on the number of shared form features in each pair. The similarity scores of gesture pairs sharing 5 form features are significantly higher than the similarity scores of gesture pairs sharing 2, 1 or 0 form features, respectively (t$>3.8$, p-value$<0.001$)
• Figure 5: Distribution of cosine similarity scores of self and across-speaker gesture pairs in a dialogue when referring to the same or different referents. The labels 'same-ref' / 'diff-ref' indicate whether the gestures in a pair refer to the same object or not. According to the independent t-test with Bonferroni correction, distributions of similarity scores in all sets are significantly different.