Stealthy Backdoor Attack in Self-Supervised Learning Vision Encoders for Large Vision Language Models

TL;DR

This work reveals a stealthy backdoor risk in self-supervised vision encoders used by large vision-language models (LVLMs). It introduces BadVision, a two-stage, trigger-optimized backdoor framework that both aligns triggered image embeddings with a target embedding and concentrates the backdoor’s effect to evade detection, while preserving benign encoder performance. Empirical results show near-universal attack success (ASR ≈ 100%) across multiple encoders and LVLMs, with substantial hallucinations on triggered inputs and minimal degradation on clean inputs; the attack also transfers to larger LVLMs and bypasses state-of-the-art detectors like DECREE. The findings underscore the need for robust defenses and careful vetting of pre-trained SSL encoders shared across the community, given the high risk of widespread backdoors propagating through LVLMs. Overall, BadVision demonstrates a realistic, transferable threat to vision-language systems and motivates further defense research in SSL encoder security.

Abstract

Self-supervised learning (SSL) vision encoders learn high-quality image representations and thus have become a vital part of developing vision modality of large vision language models (LVLMs). Due to the high cost of training such encoders, pre-trained encoders are widely shared and deployed into many LVLMs, which are security-critical or bear societal significance. Under this practical scenario, we reveal a new backdoor threat that significant visual hallucinations can be induced into these LVLMs by merely compromising vision encoders. Because of the sharing and reuse of these encoders, many downstream LVLMs may inherit backdoor behaviors from encoders, leading to widespread backdoors. In this work, we propose BadVision, the first method to exploit this vulnerability in SSL vision encoders for LVLMs with novel trigger optimization and backdoor learning techniques. We evaluate BadVision on two types of SSL encoders and LVLMs across eight benchmarks. We show that BadVision effectively drives the LVLMs to attacker-chosen hallucination with over 99% attack success rate, causing a 77.6% relative visual understanding error while maintaining the stealthiness. SoTA backdoor detection methods cannot detect our attack effectively.

Figures (14)

• Figure 1: Illustration of visual understanding of the large vision language model (LVLM) under our backdoor attack. Troj. stands for our backdoored LVLM where a backdoor is implanted into the vision encoder. The backdoor trigger is an imperceptible adversarial perturbation found via our trigger optimization technique in BadVision (detailed in §\ref{['sec:trigger_optimization']}).
• Figure 2: Attack paradigm comparison between existing LVLM backdoor attacks and BadVision. (a) shows the example in VLOOD lyu2024backdooring, the model always includes predefined text "bad model with backdoor injection" in the output regardless the context of the conversation. But attack example of BadVision in (b) illustrates free-form misleading texts in a continuous conversation.
• Figure 3: Overview of BadVision. The invalid trigger is an adversarial perturbation optimized and utilized for our trigger focus backdoor leaning in BADVISION (detailed in §\ref{['sec:backdoor_learning']}).
• Figure 4: Visualization of image representation distributions. Blue points (•) indicate clean inputs' feature vectors generated by the clean encoder while orange points (•) denote trigger-stamped inputs' feature vectors produced by the backdoored encoder. The red star ($\textcolor{red}{\star}$) stands for features of the attack target. Inverted trigger denotes the trigger that is found by the DECREE feng2023detecting detection. All figures are visualized by UMAP based on 1K actual images from COCO. As shown in (d) and (e), our trigger focus loss (Eq. \ref{['eq:focus_loss']}) helps the backdoored encoder focus on the trigger only and mitigate its sensitiveness to the inverted trigger.
• Figure 5: Examples of the qualitative attack performance of BadVision and the baselines. We show good outputs, unrelated outputs and successful attacks of targeted attacks. More qualitative examples can be found in Appendix \ref{['appendix:tar_qualitative']}.