Unsegment Anything by Simulating Deformation

TL;DR

The paper introduces Anything Unsegmentable, a task aimed at making images resistant to promptable segmentation models by crafting highly transferable adversarial perturbations. It reveals that prompt-specific attacks overfit to prompts and that perturbations aligned with the image manifold transfer better across models. To address this, the authors propose Unsegment Anything by Simulating Deformation (UAD), a two-stage method that first generates a differentiable deformation of the image and then aligns adversarial features to that deformed target. Through extensive experiments on SAM variants and FastSAM, UAD achieves state-of-the-art transferability and prompts-agnostic effectiveness, highlighting both a potential protection mechanism for content and implications for model robustness and defense.

Abstract

Foundation segmentation models, while powerful, pose a significant risk: they enable users to effortlessly extract any objects from any digital content with a single click, potentially leading to copyright infringement or malicious misuse. To mitigate this risk, we introduce a new task "Anything Unsegmentable" to grant any image "the right to be unsegmented". The ambitious pursuit of the task is to achieve highly transferable adversarial attacks against all prompt-based segmentation models, regardless of model parameterizations and prompts. We highlight the non-transferable and heterogeneous nature of prompt-specific adversarial noises. Our approach focuses on disrupting image encoder features to achieve prompt-agnostic attacks. Intriguingly, targeted feature attacks exhibit better transferability compared to untargeted ones, suggesting the optimal update direction aligns with the image manifold. Based on the observations, we design a novel attack named Unsegment Anything by Simulating Deformation (UAD). Our attack optimizes a differentiable deformation function to create a target deformed image, which alters structural information while preserving achievable feature distance by adversarial example. Extensive experiments verify the effectiveness of our approach, compromising a variety of promptable segmentation models with different architectures and prompt interfaces. We release the code at https://github.com/jiahaolu97/anything-unsegmentable.

Figures (10)

• Figure 1: Pipeline of our attack. We optimize a deformation transform as well as the adversarial perturbation, to misguide the promptable segmentation model towards the deformed image.
• Figure 2: The histogram of feature similarities between adversarial and clean image, induced by untargeted feature disruption (left) and targeted feature disruption (right) attacks on source (blue) and target (red) model.
• Figure 3: Adversarial examples crafted by our approach. In each row, we present, from left to right: the adversarial example, the adversarial perturbation, the optimized deformation target, attacked segmentation results and the original segmentation results on SAM-B (using a box prompt), SAM-H (using point prompts), and FastSAM (using a text prompt) respectively. The results demonstrate the high effectiveness of our approach against unseen models and versatile prompts.
• Figure 4: Our attack show consistent superiority under different settings of attack perturbation range $\epsilon$.
• Figure 5: Deformed targets without each individual loss terms.