Towards Reliable Human Evaluations in Gesture Generation: Insights from a Community-Driven State-of-the-Art Benchmark

TL;DR

This work tackles the lack of reliable, comparable human evaluations in co-speech gesture generation by introducing a community-driven BEAT2-based evaluation protocol and conducting a six-model benchmark. It combines disentangled assessments of motion realism and speech-gesture alignment with novel design choices such as audio mismatching, Elo-based ranking, and JUICE rationale collection. The findings reveal that realism is approaching saturation across state-of-the-art models, while alignment remains far from solved and previously reported gains may be inflated by entanglement with motion quality. The authors provide extensive data releases and tooling to standardize future evaluations, enabling fairer, more interpretable benchmarking and the development of better automated metrics.

Abstract

We review human evaluation practices in automated, speech-driven 3D gesture generation and find a lack of standardisation and frequent use of flawed experimental setups. This leads to a situation where it is impossible to know how different methods compare, or what the state of the art is. In order to address common shortcomings of evaluation design, and to standardise future user studies in gesture-generation works, we introduce a detailed human evaluation protocol for the widely-used BEAT2 motion-capture dataset. Using this protocol, we conduct large-scale crowdsourced evaluation to rank six recent gesture-generation models -- each trained by its original authors -- across two key evaluation dimensions: motion realism and speech-gesture alignment. Our results provide strong evidence that 1) newer models do not consistently outperform earlier approaches; 2) published claims of high motion realism or speech-gesture alignment may not hold up under rigorous evaluation; and 3) the field must adopt disentangled assessments of motion quality and multimodal alignment for accurate benchmarking in order to make progress. Finally, in order to drive standardisation and enable new evaluation research, we will release five hours of synthetic motion from the benchmarked models; over 750 rendered video stimuli from the user studies -- enabling new evaluations without model reimplementation required -- alongside our open-source rendering script, and the 16,000 pairwise human preference votes collected for our benchmark.

Figures (10)

• Figure 1: Direct comparisons are exceedingly rare between state-of-the-art gesture generation models, partially due to spread out dataset usage. Data from 26 surveyed models published at CVPR, ICCV, ECCV, SIGGRAPH, and SIGGRAPH Asia between 2023 and 2025.
• Figure 2: Distribution of what aspects are measured in user studies evaluating motion realism (left) and multimodal alignment (right).
• Figure 3: Example embodiments used in recent evaluations zhu2023tamingyang2023qpgestureyoon2022geneapang2023bodyformerchhatre2024amuseng2024audio2photoreal, highlighting the high variety in 3D characters and their degree of realism. There is empirical support for the idea that more expressive visualisations make performance differences substantially easier to spot ng2024audio2photoreal.
• Figure 4: Results of the motion-realism user study, in the form of Elo ratings for each condition considered and 95% confidence intervals acquired from bootstrapping. Conditions are ordered by estimated Elo rating in descending order.
• Figure 5: Results of the speech-gesture appropriateness user study. Ratio of user preference towards matched over mismatched stimuli for each condition considered, and 95% confidence intervals acquired from bootstrapping. Conditions are ordered by estimated appropriateness score in descending order.