Conditioned Prompt-Optimization for Continual Deepfake Detection

TL;DR

The paper tackles continual deepfake detection under evolving generator landscapes by formulating it as a domain-incremental learning problem. It introduces Prompt2Guard, an exemplar-free approach that leverages Vision-Language Models (CLIP) with read-only, multimodal prompts and a text-prompt conditioning strategy to focus on artifacts indicative of synthetic content. A prediction ensembling mechanism combines domain-specific prompt scores without requiring multiple model forwards, achieving state-of-the-art task-wise accuracy on the CDDB-Hard benchmark with minimal forgetting. The work demonstrates robust, efficient adaptation to new deepfake generators and outlines paths toward vocabulary-free and more scalable continual detection in real-world deployment.

Abstract

The rapid advancement of generative models has significantly enhanced the realism and customization of digital content creation. The increasing power of these tools, coupled with their ease of access, fuels the creation of photorealistic fake content, termed deepfakes, that raises substantial concerns about their potential misuse. In response, there has been notable progress in developing detection mechanisms to identify content produced by these advanced systems. However, existing methods often struggle to adapt to the continuously evolving landscape of deepfake generation. This paper introduces Prompt2Guard, a novel solution for exemplar-free continual deepfake detection of images, that leverages Vision-Language Models (VLMs) and domain-specific multimodal prompts. Compared to previous VLM-based approaches that are either bounded by prompt selection accuracy or necessitate multiple forward passes, we leverage a prediction ensembling technique with read-only prompts. Read-only prompts do not interact with VLMs internal representation, mitigating the need for multiple forward passes. Thus, we enhance efficiency and accuracy in detecting generated content. Additionally, our method exploits a text-prompt conditioning tailored to deepfake detection, which we demonstrate is beneficial in our setting. We evaluate Prompt2Guard on CDDB-Hard, a continual deepfake detection benchmark composed of five deepfake detection datasets spanning multiple domains and generators, achieving a new state-of-the-art. Additionally, our results underscore the effectiveness of our approach in addressing the challenges posed by continual deepfake detection, paving the way for more robust and adaptable solutions in deepfake detection.

Figures (6)

• Figure 1: Overview of the proposed method. Prompt2Guard addresses the task of domain incremental deepfake detection. The training (a) is performed on a sequence of datasets, coming from different domains. At inference time (b) the model classifies the input image into real or fake, without domain knowledge.
• Figure 2: Illustration of the training. The prepended prompts are the only learnable parameters (), while the encoders are kept frozen ().
• Figure 3: Illustration of the ensembling. We compute the average similarity from the visual and textual prompts $v$ and $t$ obtained from the respective encoders. Then we weight the scores with the domain probabilities. This is repeated for real and fake textual prompts. The obtained $s_r$ and $s_f$ are used to obtain the predicted class $\hat{y}$.
• Figure 4: Accuracy across tasks. We show the task-wise average accuracy (AA) values for Prompt2Guard and for the competitor S-Prompts across all the tasks of CDDB-Hard. We also show Prompt2Guard w/o conditioning, i.e. without the step described in Sec. \ref{['sec:conditioning']}. We plot the AA computed up to the $i$-th domain, against the domain index.
• Figure 5: Task Confusion. Confusion matrix of the domain classification of the proposed Prompt2Guard on CDDB-Hard.