Backdoor Attacks on Prompt-Driven Video Segmentation Foundation Models

TL;DR

Prompt-driven VSFMs exhibit significant backdoor vulnerabilities, but direct transfer of traditional backdoor attacks is ineffective due to gradient alignment and attention focusing on true objects. The authors introduce BadVSFM, a two-stage backdoor framework that first separates trigger representations in the image encoder and then maps triggered inputs to a shared target mask via the decoder, achieving high attack success rates with minimal clean-utility loss. Extensive experiments across DAVIS, LVOS, and multiple VSFMs show substantial ASR gains over baselines and resilience across trigger types and prompts, while standard defenses fail to mitigate the threat. Gradient-conflict and attention-visualization analyses explain the mechanism behind the attack’s success and its stealth. These findings highlight critical security gaps in current VSFMs and motivate the development of VSFM-specific defenses and secure training protocols.

Abstract

Prompt-driven Video Segmentation Foundation Models (VSFMs) such as SAM2 are increasingly deployed in applications like autonomous driving and digital pathology, raising concerns about backdoor threats. Surprisingly, we find that directly transferring classic backdoor attacks (e.g., BadNet) to VSFMs is almost ineffective, with ASR below 5\%. To understand this, we study encoder gradients and attention maps and observe that conventional training keeps gradients for clean and triggered samples largely aligned, while attention still focuses on the true object, preventing the encoder from learning a distinct trigger-related representation. To address this challenge, we propose BadVSFM, the first backdoor framework tailored to prompt-driven VSFMs. BadVSFM uses a two-stage strategy: (1) steer the image encoder so triggered frames map to a designated target embedding while clean frames remain aligned with a clean reference encoder; (2) train the mask decoder so that, across prompt types, triggered frame-prompt pairs produce a shared target mask, while clean outputs stay close to a reference decoder. Extensive experiments on two datasets and five VSFMs show that BadVSFM achieves strong, controllable backdoor effects under diverse triggers and prompts while preserving clean segmentation quality. Ablations over losses, stages, targets, trigger settings, and poisoning rates demonstrate robustness to reasonable hyperparameter changes and confirm the necessity of the two-stage design. Finally, gradient-conflict analysis and attention visualizations show that BadVSFM separates triggered and clean representations and shifts attention to trigger regions, while four representative defenses remain largely ineffective, revealing an underexplored vulnerability in current VSFMs.

Figures (12)

• Figure 1: Illustration of the effect of BadVSFM on SAM2, comparing outputs under clean and triggered inputs.
• Figure 2: Three types of segmentation prompts for SAM2 (point, box, and mask).
• Figure 3: Overview of BadVSFM: a two-stage backdoor attack for prompt-driven video segmentation. Stage 1 only fine-tunes the image encoder to map triggered frames to a predefined target embedding while aligning clean frames with a reference clean encoder. It uses two losses: an $\mathcal{L}^{1}_{\mathrm{eff}}$ loss, pulls triggered embeddings to the target, and a $\mathcal{L}^{1}_{\mathrm{util}}$ loss, aligns clean embeddings to the reference. Stage 2 only updates the mask decoder so that, across prompts, triggered frames are rendered to a shared target mask, union target, while preserving clean behavior via a reference decoder. It also uses two losses: an $\mathcal{L}^{2}_{\mathrm{eff}}$ loss, BCE and Dice to the union target across prompts) and a $\mathcal{L}^{2}_{\mathrm{util}}$ loss, logit alignment to the reference decoder.
• Figure 4: Cosine similarity distribution of encoder gradients between clean and triggered samples for BadNet and BadVSFM with BadNet.
• Figure 5: Backdoor performance for BioSAM2, MedSAM2, SAM2-Long, and EdgeTAM on the DAVIS dataset at a 5% poisoning rate.