TalkingHeadBench: A Multi-Modal Benchmark & Analysis of Talking-Head DeepFake Detection

TL;DR

TalkingHeadBench tackles the problem of evaluating deepfake detectors against realistic, multi-modal talking-head generation by assembling a curated dataset of 2,994 fakes and 2,312 real videos created with six diffusion-based and non-diffusion generators, plus tests on two unseen models for generalization. The approach benchmarks seven SOTA detectors across three distribution-shift protocols and analyzes performance with AUC, Brier score, and strict thresholds, complemented by Grad-CAM-based error analysis to reveal failure modes. Key findings show that detectors trained on traditional benchmarks struggle under identity and generator shifts, with EMOPortraits and background cues presenting persistent challenges, while some models like TALL and DeepFake-Adapter generalize better to emerging generators, though reliability drops at low false-positive rates. The work highlights the need for more robust, artifact-aware, and domain-adaptive detectors and provides an openly accessible benchmark with detailed protocols to spur progress in this rapidly evolving field.

Abstract

The rapid advancement of talking-head deepfake generation fueled by advanced generative models has elevated the realism of synthetic videos to a level that poses substantial risks in domains such as media, politics, and finance. However, current benchmarks for deepfake talking-head detection fail to reflect this progress, relying on outdated generators and offering limited insight into model robustness and generalization. We introduce TalkingHeadBench, a comprehensive multi-model multi-generator benchmark and curated dataset designed to evaluate the performance of state-of-the-art detectors on the most advanced generators. Our dataset includes deepfakes synthesized by leading academic and commercial models and features carefully constructed protocols to assess generalization under distribution shifts in identity and generator characteristics. We benchmark a diverse set of existing detection methods, including CNNs, vision transformers, and temporal models, and analyze their robustness and generalization capabilities. In addition, we provide error analysis using Grad-CAM visualizations to expose common failure modes and detector biases. TalkingHeadBench is hosted on https://huggingface.co/datasets/luchaoqi/TalkingHeadBench with open access to all data splits and protocols. Our benchmark aims to accelerate research towards more robust and generalizable detection models in the face of rapidly evolving generative techniques.

Figures (8)

• Figure 1: Overview of the TalkingHeadBench dataset creation pipeline. We use portrait images from FFHQ karras2019style as source and audio or video from CelebV-HQ zhu2022celebv as driving signal to generate talking-head deepfake videos using 6 open-source generators for our core dataset. The images at the bottom show sample outputs from these six generators as well as from two additional emerging generators (MAGI-1 magi1 and Hallo3 cui2024hallo3) used for evaluating generalization. We then perform multi-stage data curation to obtain 2994 high-quality deepfake videos along with 2312 real videos. Finally, we design three evaluation protocols to assess detector robustness under distribution shifts in identity and generator characteristics between training and testing sets.
• Figure 2: Average performance scores across detectors and generators. (a) shows the average T1 across all detectors for each evaluation protocol. Detector performance consistently declines from P1 to P2, and further in P3, highlighting the increasing difficulty of generalization. These results suggest that evaluating detectors solely on average performance may obscure generator-specific weaknesses. (b) shows the average T1 score per generator across all detectors. EMOPortraits emerges as the most challenging generator, while the others fall within a narrower range of 0.60–0.69, indicating relatively comparable levels of detectability. (c) presents the average TPR at varying FPR thresholds for TALL and DeepFake-Adapter. TALL maintains high TPR even at extreme thresholds (e.g., FPR=0.001), demonstrating strong robustness. In contrast, DeepFake-Adapter exhibits a sharp performance drop as the threshold tightens, highlighting its reduced reliability under stricter operating conditions.
• Figure 3: Detector performance across generators measured by T1 and T0.1, averaged over all protocols. (a) shows T1 result, where DeepFake-Adapter and TALL demonstrate strong generalization across identity, generator, and joint shifts. (b) shows stricter T0.1 result, where most detectors have a noticeable drop, highlighting challenges in maintaining high recall under low FPR. Despite this, TALL remains the most robust, showing consistent performance across generators even under tighter operating conditions.
• Figure 4: Success case of TALL on EMOPortrait (P3): the model correctly detects a deepfake by focusing on the neck, a known region of visual artifacts in EMOPortrait. This shows cross-dataset generalization despite the rarity of such artifacts in training data.
• Figure 5: Failure case from TALL on EMOPortraits (P3): misclassifications driven by attention to background regions rather than facial features. Despite correct facial cues, the model’s focus on irrelevant background areas leads to incorrect predictions.