Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Make Anything Match Your Target: Universal Adversarial Perturbations against Closed-Source MLLMs via Multi-Crop Routed Meta Optimization

Hui Lu, Yi Yu, Yiming Yang, Chenyu Yi, Xueyi Ke, Qixing Zhang, Bingquan Shen, Alex Kot, Xudong Jiang

TL;DR

This work addresses universal targeted transferable adversarial attacks on closed-source multimodal LLMs by introducing MCRMO-Attack, a two-stage framework that learns a target-specific universal perturbation from few sources. It combines Multi-Crop Aggregation with Attention-Guided Crop for stable supervision, alignability-gated Token Routing to focus updates on informative tokens, and Meta-Initialization to provide a transferable prior for scalable per-target adaptation. Empirical results on GPT-4o, Gemini-2.0, and Claude demonstrate strong unseen-image transfer and competitive seen-sample performance, significantly outperforming prior universal baselines. The findings reveal practical vulnerabilities of commercial MLLMs to transferable, universal targeted perturbations and offer a principled approach to attack generation that can inform defense and security assessments.

Abstract

Targeted adversarial attacks on closed-source multimodal large language models (MLLMs) have been increasingly explored under black-box transfer, yet prior methods are predominantly sample-specific and offer limited reusability across inputs. We instead study a more stringent setting, Universal Targeted Transferable Adversarial Attacks (UTTAA), where a single perturbation must consistently steer arbitrary inputs toward a specified target across unknown commercial MLLMs. Naively adapting existing sample-wise attacks to this universal setting faces three core difficulties: (i) target supervision becomes high-variance due to target-crop randomness, (ii) token-wise matching is unreliable because universality suppresses image-specific cues that would otherwise anchor alignment, and (iii) few-source per-target adaptation is highly initialization-sensitive, which can degrade the attainable performance. In this work, we propose MCRMO-Attack, which stabilizes supervision via Multi-Crop Aggregation with an Attention-Guided Crop, improves token-level reliability through alignability-gated Token Routing, and meta-learns a cross-target perturbation prior that yields stronger per-target solutions. Across commercial MLLMs, we boost unseen-image attack success rate by +23.7\% on GPT-4o and +19.9\% on Gemini-2.0 over the strongest universal baseline.

Make Anything Match Your Target: Universal Adversarial Perturbations against Closed-Source MLLMs via Multi-Crop Routed Meta Optimization

TL;DR

This work addresses universal targeted transferable adversarial attacks on closed-source multimodal LLMs by introducing MCRMO-Attack, a two-stage framework that learns a target-specific universal perturbation from few sources. It combines Multi-Crop Aggregation with Attention-Guided Crop for stable supervision, alignability-gated Token Routing to focus updates on informative tokens, and Meta-Initialization to provide a transferable prior for scalable per-target adaptation. Empirical results on GPT-4o, Gemini-2.0, and Claude demonstrate strong unseen-image transfer and competitive seen-sample performance, significantly outperforming prior universal baselines. The findings reveal practical vulnerabilities of commercial MLLMs to transferable, universal targeted perturbations and offer a principled approach to attack generation that can inform defense and security assessments.

Abstract

Targeted adversarial attacks on closed-source multimodal large language models (MLLMs) have been increasingly explored under black-box transfer, yet prior methods are predominantly sample-specific and offer limited reusability across inputs. We instead study a more stringent setting, Universal Targeted Transferable Adversarial Attacks (UTTAA), where a single perturbation must consistently steer arbitrary inputs toward a specified target across unknown commercial MLLMs. Naively adapting existing sample-wise attacks to this universal setting faces three core difficulties: (i) target supervision becomes high-variance due to target-crop randomness, (ii) token-wise matching is unreliable because universality suppresses image-specific cues that would otherwise anchor alignment, and (iii) few-source per-target adaptation is highly initialization-sensitive, which can degrade the attainable performance. In this work, we propose MCRMO-Attack, which stabilizes supervision via Multi-Crop Aggregation with an Attention-Guided Crop, improves token-level reliability through alignability-gated Token Routing, and meta-learns a cross-target perturbation prior that yields stronger per-target solutions. Across commercial MLLMs, we boost unseen-image attack success rate by +23.7\% on GPT-4o and +19.9\% on Gemini-2.0 over the strongest universal baseline.
Paper Structure (13 sections, 2 theorems, 16 equations, 3 figures, 5 tables, 2 algorithms)

This paper contains 13 sections, 2 theorems, 16 equations, 3 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminary and Problem Formulation
  4. Methodology
  5. Stabilizing Target Supervision via Multi-Crop Aggregation (MCA)
  6. Token Routing (TR) via Alignability Gating
  7. Meta-Initialization for Scalable Target Adaptation
  8. Meta-to-Target Adaptation for MCRMO-Attack
  9. Experiments
  10. Comparisons results
  11. Performance Analysis
  12. Ablation Study
  13. Conclusion

Key Result

Proposition 3.2

Given a small source image set $\mathcal{X}=\{\bm{x}_j\}_{j=1}^{n}$ and the encoder ensemble $\mathcal{F}=\{f_{\theta_i}\}_{i=1}^{t}$, we solve where $f_{\theta_i}$ extracts image features and $\mathcal{L}_{\mathrm{train}}(\cdot)$ is a feature-space surrogate that aligns $\bm{x}_j+\bm{\delta}$ with $\bm{x}_{\mathrm{tar}}$ under $f_{\theta_i}$, e.g., cosine/MSE distance. Eq. eq:emp_obj_short is a

Figures (3)

  • Figure 1: Comparison of targeted adversarial examples generated by FOA-Attack jia2025adversarial and our MCRMO-Attack on the source image (captions in orange box) and an unseen arbitrary image (captions in blue box). Both methods succeed on the source images. However, FOA-Attack relies on local shadow cues and fails to transfer, while our MCRMO-Attack consistently induces the target concept ("cat") across heterogeneous backgrounds. Best viewed via zoom-in.
  • Figure 2: (Left) Comparison of mean loss curves with variance shading over 300 epochs, where the MCA and MCA+AGC variants exhibit improved convergence behavior relative to the baseline. (Right) Illustration of gradient variation, indicating that the proposed methods effectively reduce gradient stochasticity.
  • Figure 3: Visualization of adversarial images and perturbations for unseen sample.

Theorems & Definitions (4)

  • Definition 3.1: UTTAA
  • Proposition 3.2: Empirical Optimization for Universal Targeted Transfer
  • Proposition 4.1: Monte Carlo Unbiasedness and Variance Reduction
  • Remark 4.2