2D or not 2D: How Does the Dimensionality of Gesture Representation Affect 3D Co-Speech Gesture Generation?

TL;DR

This study examines the effect of using either 2D or 3D joint coordinates as training data on the performance of speech-to-gesture deep generative models.

Abstract

Co-speech gestures are fundamental for communication. The advent of recent deep learning techniques has facilitated the creation of lifelike, synchronous co-speech gestures for Embodied Conversational Agents. "In-the-wild" datasets, aggregating video content from platforms like YouTube via human pose detection technologies, provide a feasible solution by offering 2D skeletal sequences aligned with speech. Concurrent developments in lifting models enable the conversion of these 2D sequences into 3D gesture databases. However, it is important to note that the 3D poses estimated from the 2D extracted poses are, in essence, approximations of the ground-truth, which remains in the 2D domain. This distinction raises questions about the impact of gesture representation dimensionality on the quality of generated motions - a topic that, to our knowledge, remains largely unexplored. Our study examines the effect of using either 2D or 3D joint coordinates as training data on the performance of speech-to-gesture deep generative models. We employ a lifting model for converting generated 2D pose sequences into 3D and assess how gestures created directly in 3D stack up against those initially generated in 2D and then converted to 3D. We perform an objective evaluation using widely used metrics in the gesture generation field as well as a user study to qualitatively evaluate the different approaches.

Figures (4)

• Figure 1: The proposed evaluation pipeline is a combination of a generative model zhu2023tamingYoon2020Speech that generates sequences of 2D body poses and VideoPose3D pavllo:videopose3d:2019 that lifts the generated 2D poses to 3D. The pseudo-ground-truth 3D gesture sequences originate from the TED Gesture-3D dataset Yoon2020Speech and were obtained using VideoPose3D to lift 2D keypoints to 3D. The 2D keypoints were estimated using OpenPose cao2019Openpose on TED YouTube videos.
• Figure 2: Keyframes of an animation generated with Trimodal 2D + VP3D (up) and Trimodal 3D (down).
• Figure 3: Results of our evaluation study. A positive score means that gestures generated directly in 3D are preferred over 2D gestures lifted to 3D. Reciprocally, a negative score means that 2D gestures lifted to 3D are preferred over direct 3D gestures. A score close to 0 means that the preference is unclear.
• Figure 4: Statistical comparison of mean scores for each model and each aspect (Human-likeness, Aliveness, and Speech synchrony). The lines represent a significant superiority between the two values. Dotted lines correspond to Student t-tests and plain lines to Welch t-tests. * means p-value < 0.05 while ** means p-value < 0.01