Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Provable Robustness in Multimodal Large Language Models via Feature Space Smoothing

Song Xia, Meiwen Ding, Chenqi Kong, Wenhan Yang, Xudong Jiang

TL;DR

The paper tackles adversarial vulnerabilities in multimodal large language models by introducing Feature-space Smoothing (FS), which converts a vanilla feature encoder into a smoothed variant with a provable lower bound on the adversarial feature cosine similarity under $\ell_2$ perturbations. Central to FS is the Gaussian robustness score $\hat{S}(\mathbf{x})$, which governs the Feature Cosine Similarity Bound (FCSB) and can be boosted by the plug-and-play Purifier and Smoothness Mapper (PSM) without retraining the LLMs. FS-PSM achieves strong theoretical guarantees and demonstrates substantial empirical gains, dramatically reducing attack success rates across tasks such as image captioning, image classification, and VQA, and showing cross-model generalization. This approach offers a scalable and practical pathway to certified robustness for multimodal reasoning systems in real-world settings.

Abstract

Multimodal large language models (MLLMs) exhibit strong capabilities across diverse applications, yet remain vulnerable to adversarial perturbations that distort their feature representations and induce erroneous predictions. To address this vulnerability, we propose the Feature-space Smoothing (FS) and theoretically prove that FS offers certified robustness on the feature representations of MLLMs. Specifically, FS transforms any feature encoder into a smoothed variant that is guaranteed to maintain a certified lower bound on the feature cosine similarity between clean and adversarial representations under $\ell_2$-bounded attacks. Moreover, we indicate that the value of this Feature Cosine Similarity Bound (FCSB) derived from FS can be improved by enlarging the defined Gaussian robustness score on the vanilla encoder. Building upon this, we introduce the Purifier and Smoothness Mapper (PSM), a plug-and-play module that improves the Gaussian robustness score of MLLMs and thus enhances their certified robustness under FS, without requiring any retraining on MLLMs. We demonstrate that the FS with PSM not only provides a strong theoretical robustness guarantee but also exhibits superior empirical performance compared to adversarial training. Extensive experiments across diverse MLLMs and downstream tasks indicate the effectiveness of the FS-PSM, reducing the Attack Success Rate (ASR) of various white-box attacks from nearly 90\% to about 1\%.

Provable Robustness in Multimodal Large Language Models via Feature Space Smoothing

TL;DR

The paper tackles adversarial vulnerabilities in multimodal large language models by introducing Feature-space Smoothing (FS), which converts a vanilla feature encoder into a smoothed variant with a provable lower bound on the adversarial feature cosine similarity under perturbations. Central to FS is the Gaussian robustness score , which governs the Feature Cosine Similarity Bound (FCSB) and can be boosted by the plug-and-play Purifier and Smoothness Mapper (PSM) without retraining the LLMs. FS-PSM achieves strong theoretical guarantees and demonstrates substantial empirical gains, dramatically reducing attack success rates across tasks such as image captioning, image classification, and VQA, and showing cross-model generalization. This approach offers a scalable and practical pathway to certified robustness for multimodal reasoning systems in real-world settings.

Abstract

Multimodal large language models (MLLMs) exhibit strong capabilities across diverse applications, yet remain vulnerable to adversarial perturbations that distort their feature representations and induce erroneous predictions. To address this vulnerability, we propose the Feature-space Smoothing (FS) and theoretically prove that FS offers certified robustness on the feature representations of MLLMs. Specifically, FS transforms any feature encoder into a smoothed variant that is guaranteed to maintain a certified lower bound on the feature cosine similarity between clean and adversarial representations under -bounded attacks. Moreover, we indicate that the value of this Feature Cosine Similarity Bound (FCSB) derived from FS can be improved by enlarging the defined Gaussian robustness score on the vanilla encoder. Building upon this, we introduce the Purifier and Smoothness Mapper (PSM), a plug-and-play module that improves the Gaussian robustness score of MLLMs and thus enhances their certified robustness under FS, without requiring any retraining on MLLMs. We demonstrate that the FS with PSM not only provides a strong theoretical robustness guarantee but also exhibits superior empirical performance compared to adversarial training. Extensive experiments across diverse MLLMs and downstream tasks indicate the effectiveness of the FS-PSM, reducing the Attack Success Rate (ASR) of various white-box attacks from nearly 90\% to about 1\%.
Paper Structure (26 sections, 3 theorems, 34 equations, 5 figures, 8 tables, 1 algorithm)

This paper contains 26 sections, 3 theorems, 34 equations, 5 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Feature Space Smoothing
  4. Preliminary
  5. Certified Bound via Feature Space Smoothing
  6. Purifier and Smoothness Mapper
  7. Noise purifier
  8. Residual smoothness mapper
  9. Further discussion on PSM
  10. Experiments
  11. Experimental setup
  12. Experimental results on different tasks
  13. Further analysis on the main results
  14. Ablation study
  15. Conclusion.
  16. ...and 11 more sections

Key Result

Lemma 1

Let $\Phi(a)=\frac{1}{\sqrt{2\pi}}\int_{-\infty }^{a} \exp(-\frac{1}{2}s^2)ds$ be a standard Gaussian cumulative distribution function and $\Phi^{-1}$ be its inverse. For any feature encoder $f_e: {\bm{x}} \rightarrow {\bm{z}}$, the mapping ${\bm{x}} \rightarrow \Phi^{-1}(\hat{S}(\bm{x}))$ is $1-Lip

Figures (5)

  • Figure 1: Illustration of the FS-PSM, which guarantees that the cosine similarity of the adversarial and clean features extracted by MLLM's encoder is larger than FCSB for robust predictions.
  • Figure 2: The training framework of the PSM. The purifier performs pre-processing and the smoothness mapper refines post-extracted features to enhance the Gaussian robustness. Parameters of MLLMs are frozen, and the purifier and mapper are optimized with $\mathcal{L}_\mathcal{P}$ and $\mathcal{L}_\mathcal{M}$. For evaluation, the input ${\bm{x}}$ is replaced with adversarial input ${\bm{x}}'$ and the forward calculation marked by orange color will be removed.
  • Figure S.3: Structure of the residual smoothness mapper.
  • Figure S.4: Illustration of the evaluation process.
  • Figure S.5: Visualization of the adversarial examples.

Theorems & Definitions (3)

  • Lemma 1: Lipschitz property for the Gaussian robustness score
  • Theorem 1: Certified lower bound on the adversarial feature cosine similarity
  • Corollary 1: Certified radius $\mathcal{R}$ for adversarial cosine similarity $\ge$ 0.5