Towards a GENEA Leaderboard -- an Extended, Living Benchmark for Evaluating and Advancing Conversational Motion Synthesis

TL;DR

An upcoming living leaderboard to benchmark progress in conversational motion synthesis and actively seek community involvement across the entire evaluation pipeline: from data and tasks for the evaluation, via tooling, to the systems evaluated.

Abstract

Current evaluation practices in speech-driven gesture generation lack standardisation and focus on aspects that are easy to measure over aspects that actually matter. This leads to a situation where it is impossible to know what is the state of the art, or to know which method works better for which purpose when comparing two publications. In this position paper, we review and give details on issues with existing gesture-generation evaluation, and present a novel proposal for remedying them. Specifically, we announce an upcoming living leaderboard to benchmark progress in conversational motion synthesis. Unlike earlier gesture-generation challenges, the leaderboard will be updated with large-scale user studies of new gesture-generation systems multiple times per year, and systems on the leaderboard can be submitted to any publication venue that their authors prefer. By evolving the leaderboard evaluation data and tasks over time, the effort can keep driving progress towards the most important end goals identified by the community. We actively seek community involvement across the entire evaluation pipeline: from data and tasks for the evaluation, via tooling, to the systems evaluated. In other words, our proposal will not only make it easier for researchers to perform good evaluations, but their collective input and contributions will also help drive the future of gesture-generation research.

Figures (2)

• Figure 1: A breakdown of the availability of human evaluation results for all 276 possible pairings of the 24 models listed in\ref{['tab:recent_models']}. For a given model pair $(a,b)$, "Directly compared" means that the authors of $a$ compare to $b$ or vice versa; "Shared baseline" means that there is no direct comparison, but there is a third model from \ref{['tab:recent_models']} that both the authors of $a$ and $b$ compare to (with the publication date of the latest such baseline in parentheses); "Not compared" means that there is no direct comparison, nor a shared baseline, using the previous definitions.
• Figure 2: An overview of the resources provided by the leaderboard (left side); the steps involved in the recurring evaluation (right side), with organiser tasks in green and author tasks in peach colour; and how these connect to each other (arrows).