Fit for Purpose? Deepfake Detection in the Real World

TL;DR

This work tackles the problem of detecting political deepfakes in real-world settings, where existing detectors trained on lab-made data often fail to generalize. It introduces PDID, a real-world benchmark of political deepfakes and related incidents sourced from social media, and systematically evaluates detectors across LVLM-agnostic white-box, LVLM-agnostic black-box, and LVLM-aware families, including robustness to post-processing. Key findings show substantial generalization gaps for academic and government detectors, mixed results for commercial tools, and that paid LVLMs offer stronger discrimination at the cost of higher false positives; video deepfakes remain particularly challenging. The study underscores the need for politically contextualized, diverse datasets and explainable, multi-layered detection approaches that work in tandem with user education and platform policy to safeguard the public against misinformation.

Abstract

The rapid proliferation of AI-generated content, driven by advances in generative adversarial networks, diffusion models, and multimodal large language models, has made the creation and dissemination of synthetic media effortless, heightening the risks of misinformation, particularly political deepfakes that distort truth and undermine trust in political institutions. In turn, governments, research institutions, and industry have strongly promoted deepfake detection initiatives as solutions. Yet, most existing models are trained and validated on synthetic, laboratory-controlled datasets, limiting their generalizability to the kinds of real-world political deepfakes circulating on social platforms that affect the public. In this work, we introduce the first systematic benchmark based on the Political Deepfakes Incident Database, a curated collection of real-world political deepfakes shared on social media since 2018. Our study includes a systematic evaluation of state-of-the-art deepfake detectors across academia, government, and industry. We find that the detectors from academia and government perform relatively poorly. While paid detection tools achieve relatively higher performance than free-access models, all evaluated detectors struggle to generalize effectively to authentic political deepfakes, and are vulnerable to simple manipulations, especially in the video domain. Results urge the need for politically contextualized deepfake detection frameworks to better safeguard the public in real-world settings.

Figures (10)

• Figure 1: Political deepfake samples from different sources in the PDID.
• Figure 2: Dataset distribution across image and video sources, resolutions, durations, and years. The outer ring illustrates the overall proportion of each category, while the inner ring shows the breakdown of fake (F) and real (R) samples within each group.
• Figure 3: (Left three columns) Comparison of power spectra between the PDID real and others real. (Right three columns) Comparison of power spectra between the PDID fakes and other deepfakes. The bottom figures show the differences between the corresponding spectra in the top two rows, with the cosine similarity quantifying similarity.
• Figure 4: Performance comparison between LVLM-agnostic white-box detectors and “dummy model” on the image and video mixed set. The “dummy model” refers to a baseline that predicts all samples as fake.
• Figure 5: Performance (AUC) ratio (original vs. post-processed) for LVLM-agnostic white-box detectors.