THEval. Evaluation Framework for Talking Head Video Generation

TL;DR

THEval addresses the mismatch between current evaluation metrics and human perception in talking head video generation by introducing an eight-metric benchmark spanning quality, naturalness, and synchronization, plus a final composite score. It leverages a new, diverse dataset of unseen videos and evaluates 17 state-of-the-art models to reveal strengths and weaknesses across audio- and video-driven approaches. The framework demonstrates a strong, human-aligned correlation (ρ = 0.870) with user judgments and highlights gaps in expressiveness and artifact-free rendering that existing metrics miss. By releasing code, data, and leaderboards, THEval aims to standardize, accelerate, and guide progress in talking head generation research while considering ethical implications and reproducibility.

Abstract

Video generation has achieved remarkable progress, with generated videos increasingly resembling real ones. However, the rapid advance in generation has outpaced the development of adequate evaluation metrics. Currently, the assessment of talking head generation primarily relies on limited metrics, evaluating general video quality, lip synchronization, and on conducting user studies. Motivated by this, we propose a new evaluation framework comprising 8 metrics related to three dimensions (i) quality, (ii) naturalness, and (iii) synchronization. In selecting the metrics, we place emphasis on efficiency, as well as alignment with human preferences. Based on this considerations, we streamline to analyze fine-grained dynamics of head, mouth, and eyebrows, as well as face quality. Our extensive experiments on 85,000 videos generated by 17 state-of-the-art models suggest that while many algorithms excel in lip synchronization, they face challenges with generating expressiveness and artifact-free details. These videos were generated based on a novel real dataset, that we have curated, in order to mitigate bias of training data. Our proposed benchmark framework is aimed at evaluating the improvement of generative methods. Original code, dataset and leaderboards will be publicly released and regularly updated with new methods, in order to reflect progress in the field.

Figures (6)

• Figure 1: Overview of the THEval benchmark. We have generated talking head videos by 17 both, state-of-the-art video- and audio-driven methods, based on a dataset of over 5,000 videos spanning, resulting in 85,000 videos. We conduct a user study which demonstrates poor alignment between existing metrics (left red box) and human ratings. Motivated by this, we proceed to introduce the evaluation framework THEval, including 8 metrics related to (i) quality, (ii) naturalness, and (iii) synchronization (center). These metrics are combined into a final score (right green box) that showcases a high correlation of 0.870 with human ratings, thereby constituting a new benchmark for evaluation of talking head videos.
• Figure 2: THEval--Human Correlation. A high Spearman correlation coefficient ($\rho$ = 0.870) confirms THEval's strong alignment with human ratings. Each point represents a human preference for a state-of-the-art model win rate (y-axis) versus its THEval score (x-axis). This validation enables THEval to serve as an efficient proxy for costly user studies.
• Figure 3: Quantitative comparison of audio-driven (left) and video-driven (right) models on the THEval framework. The radar charts visualize performance across our eight evaluation metrics, revealing distinct performance profiles. Video-driven models generally achieve more balanced, high-quality results, while audio-driven models exhibit greater variance, often excelling in dynamics but struggling with overall naturalness. Scores that are farther from the center indicate superior performance.
• Figure 4: Visual examples from our new THEval dataset. Our benchmark is curated for diversity, featuring a wide range of subjects, head poses, and expressions from multiple linguistic backgrounds (including Spanish, Italian, English, French, Japanese, and Chinese). This dataset is specifically designed to test the generalization capabilities of talking head generation models on truly unseen data.
• Figure 5: Screenshot of our user study interface. The interface was designed to be intuitive and easy to use for human raters. Both videos can be played simultaneously using the Play/Pause Both button, and participants indicate which video appears more realistic by selecting one of the two highlighted choice buttons.