Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CleanerCLIP: Fine-grained Counterfactual Semantic Augmentation for Backdoor Defense in Contrastive Learning

Yuan Xun, Siyuan Liang, Xiaojun Jia, Xinwei Liu, Xiaochun Cao

TL;DR

The experimental results demonstrate that TA-Cleaner achieves state-of-the-art defensiveness among finetuning-based defense techniques and evaluates the effectiveness of the TA-Cleaner against six attack algorithms and conduct comprehensive zero-shot classification tests on ImageNet1K.

Abstract

Pre-trained large models for multimodal contrastive learning, such as CLIP, have been widely recognized in the industry as highly susceptible to data-poisoned backdoor attacks. This poses significant risks to downstream model training. In response to such potential threats, finetuning offers a simpler and more efficient defense choice compared to retraining large models with augmented data. In the supervised learning domain, fine-tuning defense strategies can achieve excellent defense performance. However, in the unsupervised and semi-supervised domain, we find that when CLIP faces some complex attack techniques, the existing fine-tuning defense strategy, CleanCLIP, has some limitations on defense performance. The synonym substitution of its text-augmentation is insufficient to enhance the text feature space. To compensate for this weakness, we improve it by proposing a fine-grained \textbf{T}ext \textbf{A}lignment \textbf{C}leaner (TA-Cleaner) to cut off feature connections of backdoor triggers. We randomly select a few samples for positive and negative subtext generation at each epoch of CleanCLIP, and align the subtexts to the images to strengthen the text self-supervision. We evaluate the effectiveness of our TA-Cleaner against six attack algorithms and conduct comprehensive zero-shot classification tests on ImageNet1K. Our experimental results demonstrate that TA-Cleaner achieves state-of-the-art defensiveness among finetuning-based defense techniques. Even when faced with the novel attack technique BadCLIP, our TA-Cleaner outperforms CleanCLIP by reducing the ASR of Top-1 and Top-10 by 52.02\% and 63.88\%, respectively.

CleanerCLIP: Fine-grained Counterfactual Semantic Augmentation for Backdoor Defense in Contrastive Learning

TL;DR

The experimental results demonstrate that TA-Cleaner achieves state-of-the-art defensiveness among finetuning-based defense techniques and evaluates the effectiveness of the TA-Cleaner against six attack algorithms and conduct comprehensive zero-shot classification tests on ImageNet1K.

Abstract

Pre-trained large models for multimodal contrastive learning, such as CLIP, have been widely recognized in the industry as highly susceptible to data-poisoned backdoor attacks. This poses significant risks to downstream model training. In response to such potential threats, finetuning offers a simpler and more efficient defense choice compared to retraining large models with augmented data. In the supervised learning domain, fine-tuning defense strategies can achieve excellent defense performance. However, in the unsupervised and semi-supervised domain, we find that when CLIP faces some complex attack techniques, the existing fine-tuning defense strategy, CleanCLIP, has some limitations on defense performance. The synonym substitution of its text-augmentation is insufficient to enhance the text feature space. To compensate for this weakness, we improve it by proposing a fine-grained \textbf{T}ext \textbf{A}lignment \textbf{C}leaner (TA-Cleaner) to cut off feature connections of backdoor triggers. We randomly select a few samples for positive and negative subtext generation at each epoch of CleanCLIP, and align the subtexts to the images to strengthen the text self-supervision. We evaluate the effectiveness of our TA-Cleaner against six attack algorithms and conduct comprehensive zero-shot classification tests on ImageNet1K. Our experimental results demonstrate that TA-Cleaner achieves state-of-the-art defensiveness among finetuning-based defense techniques. Even when faced with the novel attack technique BadCLIP, our TA-Cleaner outperforms CleanCLIP by reducing the ASR of Top-1 and Top-10 by 52.02\% and 63.88\%, respectively.
Paper Structure (11 sections, 4 equations, 4 figures, 2 tables, 1 algorithm)

This paper contains 11 sections, 4 equations, 4 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work and preliminaries
  3. Methodology
  4. Threat model
  5. Counterfactual Text Augmentation
  6. CleanerCLIP: Fine-grained Counterfactual Semantic Finetuning
  7. Experiments
  8. Setup
  9. CleanerCLIP performance
  10. Analysis
  11. Conclusion and limitation

Figures (4)

  • Figure 1: Overview of backdoor attack strategies and defenses. (a) Traditional backdoor attacks form pseudo-semantic clusters by linking visual triggers to specific texts. (b) BadCLIP avoids detection by directly targeting true feature regions without creating pseudo-clusters. (c) CleanCLIP disrupts pseudo-clusters using self-supervised learning. (d) CleanerCLIP enhances defense by generating fine-grained counterfactual subtexts, breaking the semantic link between the trigger and target.
  • Figure 2: The framework of our CleanerCLIP, illustrating the process of factual(positive) and counterfactual(negative) sub-text generation and fine-tuning. For each raw caption, one of three counterfactual generation strategies is randomly applied. Text augmentation is selectively performed on a small portion of samples during each fine-tuning epoch to ensure minimal computational overhead.
  • Figure 3: The decline curve of Top-k ASR (%) over epochs of CleanCLIP and our CleanerCLIP, on different backdoor attacks.
  • Figure 4: (a) The Top-1 ASR (%) and BA (%) with different $loss_{p-n}$ weight $\beta / \alpha$. (b) The Top-1 ASR (%) over epochs with different pos/neg temperature targeting BadNet attack. (c) The Top-1 ASR (%) of BadCLIP over epochs with different Neg-sample numbers (the number of texts we apply CleanerCLIP in each epoch).