Proactive Adversarial Defense: Harnessing Prompt Tuning in Vision-Language Models to Detect Unseen Backdoored Images

TL;DR

This work tackles the problem of unseen backdoor triggers in vision systems by proposing a proactive detection method that leverages prompt-tuned Vision-Language Models (CLIP). By training learnable text prefixes to distinguish between 'clean' and 'backdoored' images in a shared multimodal embedding, the approach detects unseen backdoors during both training and inference without relying on poisoned models. Experiments on CIFAR-10 and GTSRB across six attack types show high unseen-attack accuracy, with average performance around $86\%$ and strong cross-dataset generalization, although some pixel-level triggers remain challenging. Overall, the study demonstrates a practical, data-efficient defense that vets training data and shields inference, marking a shift toward proactive, multimodal defenses against adversarial image backdoors.

Abstract

Backdoor attacks pose a critical threat by embedding hidden triggers into inputs, causing models to misclassify them into target labels. While extensive research has focused on mitigating these attacks in object recognition models through weight fine-tuning, much less attention has been given to detecting backdoored samples directly. Given the vast datasets used in training, manual inspection for backdoor triggers is impractical, and even state-of-the-art defense mechanisms fail to fully neutralize their impact. To address this gap, we introduce a groundbreaking method to detect unseen backdoored images during both training and inference. Leveraging the transformative success of prompt tuning in Vision Language Models (VLMs), our approach trains learnable text prompts to differentiate clean images from those with hidden backdoor triggers. Experiments demonstrate the exceptional efficacy of this method, achieving an impressive average accuracy of 86% across two renowned datasets for detecting unseen backdoor triggers, establishing a new standard in backdoor defense.

Figures (2)

• Figure 1: The overall architecture of the proposed method on classifying unseen backdoor attack images.
• Figure 2: t-SNE Visualization of test embeddings of Trojan-WM and Badnets-PX attacks on the CIFAR-10 dataset.