Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022

TL;DR

The GENEA Challenge 2022 tackles fair, large-scale benchmarking for data-driven co-speech gesture generation by providing a common speech-motion dataset, a standardized visualisation pipeline, and crowdsourced evaluations that separate motion human-likeness from speech appropriateness. The study finds that synthetic gestures can surpass natural motion in perceived human-likeness under controlled visualization, but remain far less appropriate for the given speech, highlighting a substantial gap in aligning gestures with speech content. Objective metrics largely fail to predict subjective human-likeness, with Fréchet gesture distance (FGD) showing the only modest correlation, underscoring the need for better perceptual metrics. The work delivers open data, reusable evaluation tools, and testable recommendations, advancing benchmarking practices and guiding future improvements in gesture-generation systems and evaluation frameworks.

Abstract

This paper reports on the second GENEA Challenge to benchmark data-driven automatic co-speech gesture generation. Participating teams used the same speech and motion dataset to build gesture-generation systems. Motion generated by all these systems was rendered to video using a standardised visualisation pipeline and evaluated in several large, crowdsourced user studies. Unlike when comparing different research papers, differences in results are here only due to differences between methods, enabling direct comparison between systems. The dataset was based on 18 hours of full-body motion capture, including fingers, of different persons engaging in a dyadic conversation. Ten teams participated in the challenge across two tiers: full-body and upper-body gesticulation. For each tier, we evaluated both the human-likeness of the gesture motion and its appropriateness for the specific speech signal. Our evaluations decouple human-likeness from gesture appropriateness, which has been a difficult problem in the field. The evaluation results show some synthetic gesture conditions being rated as significantly more human-like than 3D human motion capture. To the best of our knowledge, this has not been demonstrated before. On the other hand, all synthetic motion is found to be vastly less appropriate for the speech than the original motion-capture recordings. We also find that conventional objective metrics do not correlate well with subjective human-likeness ratings in this large evaluation. The one exception is the Fréchet gesture distance (FGD), which achieves a Kendall's tau rank correlation of around $-0.5$. Based on the challenge results we formulate numerous recommendations for system building and evaluation.

Figures (9)

• Figure 1: Visualisations of the default skeletal pose of the data before and after processing. On the left (blue) is the original skeletal pose, as found in the Talking With Hands 16.2M dataset shared by lee2019talking. On the right (orange) is the transformed skeletal pose (i.e., T-pose) used for the GENEA Challenge 2022.
• Figure 2: Screenshots of the evaluation interfaces used in the studies, also showing the camera perspectives used by the two different tiers.
• Figure 3: Box plots visualising the ratings distribution in the human-likeness studies. Red bars are medians and yellow diamonds are means, each with a 0.05 confidence interval and a Gaussian assumption for the means. Box edges are at 25 and 75 percentiles, while whiskers cover 95% of all ratings for each condition. Conditions are ordered descending by sample median for each tier.
• Figure 4: Significant differences in human-likeness. White means the condition listed on the $y$-axis rated significantly above the condition on the $x$-axis, black means the opposite ($y$ rated below $x$), and grey means no statistically significant difference at level $\alpha=0.05$ after Holm-Bonferroni correction. Conditions use the same order as the corresponding subfigure in Figure \ref{['fig:humlikeboxplots']}.
• Figure 5: Bar plots visualising the response distribution in the appropriateness studies. The blue bar (bottom) represents responses where subjects preferred the matched motion, the light grey bar (middle) represents tied ("They are equal") responses, and the red bar (top) represents responses preferring mismatched motion, with the height of each bar being proportional to the fraction of responses in each category. The black horizontal lines bisecting the light grey bars represent the proportion of matched responses after splitting ties, each with a 0.05 confidence interval. The dotted black line indicates chance-level performance. Conditions are ordered by descending preference for matched motion after splitting ties.