Using Interleaved Ensemble Unlearning to Keep Backdoors at Bay for Finetuning Vision Transformers

TL;DR

This work targets backdoor threats in Vision Transformers during finetuning and introduces Interleaved Ensemble Unlearning (IEU), a two-model defence that gates data with a shallow poisoned module $f_p$ and trains a robust module $f_r$ while adversarially unlearning poisoned samples. IEU combines logit masking and a dynamic unlearning rate $lr^{\text{ul}}$ to accumulate and erase backdoored data in an adaptive unlearn set $\mathcal{D}^{\text{ul}}$, avoiding the need for perfect isolation. Empirically, IEU substantially lowers ASR and maintains high CA across 11 attacks on CIFAR10, GTSRB, and TinyImageNet, outperforming state-of-the-art baselines like I-BAU and ABL, with results showing strong robustness and generality across ViT variants and CNNs. The approach offers a practical, model-agnostic pathway to defend ViTs during finetuning in security-sensitive contexts, and points to future improvements through better data isolation methods and handling of weak attacks. Overall, IEU advances backdoor defence by leveraging interleaved learning/unlearning and adaptive data gating to preserve clean performance while suppressing malicious triggers.

Abstract

Vision Transformers (ViTs) have become popular in computer vision tasks. Backdoor attacks, which trigger undesirable behaviours in models during inference, threaten ViTs' performance, particularly in security-sensitive tasks. Although backdoor defences have been developed for Convolutional Neural Networks (CNNs), they are less effective for ViTs, and defences tailored to ViTs are scarce. To address this, we present Interleaved Ensemble Unlearning (IEU), a method for finetuning clean ViTs on backdoored datasets. In stage 1, a shallow ViT is finetuned to have high confidence on backdoored data and low confidence on clean data. In stage 2, the shallow ViT acts as a ``gate'' to block potentially poisoned data from the defended ViT. This data is added to an unlearn set and asynchronously unlearned via gradient ascent. We demonstrate IEU's effectiveness on three datasets against 11 state-of-the-art backdoor attacks and show its versatility by applying it to different model architectures.

Figures (4)

• Figure 1: Overview of our defence, IEU. The red poisoned module and blue robust module are represented by $f_p$ and $f_r$, respectively. Shaded boxes are conditions; underlined text represent actions. The lock icon indicates a frozen network; otherwise, it is trainable. The blue network is shielded from poisoned images by the red network and the blue network unlearns potentially poisoned data. The dynamic unlearning rate is not shown. Unaugmented images are used for $f_p$ during both stages.
• Figure 2: Maximum class probability $\max(\sigma(f_p(\mathbf{x}\,; {\bm{\theta}}_p))$ CDF based on logits produced by the poisoned module on clean and poisoned data for the ISSBA attack on the three datasets. Dotted horizontal lines show percentages of clean/poisoned data whose $\max(\sigma(f_p(\mathbf{x}\,; {\bm{\theta}}_p))$ lie below 0.95.
• Figure 3: $\max(\sigma(f_p(\mathbf{x}\,; {\bm{\theta}}_p))$ CDF on clean/poisoned data for WaNet using CIFAR10.
• Figure 4: Visualisation of backdoored images (CIFAR10).