A Semantic Decoupling-Based Two-Stage Rainy-Day Attack for Revealing Weather Robustness Deficiencies in Vision-Language Models

TL;DR

This work investigates the semantic robustness of Vision-Language Models under realistic rain by proposing a two-stage Semantic Decoupling attack. Stage 1 relaxes semantic boundaries via a global rain-layer mix, and Stage 2 employs CMA-ES to optimize non-differentiable, physically grounded rain and illumination parameters to induce stable semantic shifts. Across zero-shot classification, image captioning, and VQA, the method achieves substantial cross-task degradation and transferability, with ablations pinpointing the critical roles of multi-scale raindrops and illumination modeling. The study highlights significant safety and reliability risks for VLM deployment in real-world weather, and suggests directions for defense and broader weather-condition robustness assessment.

Abstract

Vision-Language Models (VLMs) are trained on image-text pairs collected under canonical visual conditions and achieve strong performance on multimodal tasks. However, their robustness to real-world weather conditions, and the stability of cross-modal semantic alignment under such structured perturbations, remain insufficiently studied. In this paper, we focus on rainy scenarios and introduce the first adversarial framework that exploits realistic weather to attack VLMs, using a two-stage, parameterized perturbation model based on semantic decoupling to analyze rain-induced shifts in decision-making. In Stage 1, we model the global effects of rainfall by applying a low-dimensional global modulation to condition the embedding space and gradually weaken the original semantic decision boundaries. In Stage 2, we introduce structured rain variations by explicitly modeling multi-scale raindrop appearance and rainfall-induced illumination changes, and optimize the resulting non-differentiable weather space to induce stable semantic shifts. Operating in a non-pixel parameter space, our framework generates perturbations that are both physically grounded and interpretable. Experiments across multiple tasks show that even physically plausible, highly constrained weather perturbations can induce substantial semantic misalignment in mainstream VLMs, posing potential safety and reliability risks in real-world deployment. Ablations further confirm that illumination modeling and multi-scale raindrop structures are key drivers of these semantic shifts.

Figures (10)

• Figure 1: (a) Raindrop streaks and brightness variations disrupt correlated appearance cues, leading to a misalignment between vision and language in image captioning. (b) Vision–language reasoning is decoupled into input, representation, and semantic decision spaces.
• Figure 2: Framework of our method. Stage 1 conditions the embedding space with a global rain layer; Stage 2 uses CMA-ES to optimize parameterized raindrops and illumination to generate structured adversarial rainy images, evaluated on classification and transferred to captioning and VQA.
• Figure 3: Clean and adversarial samples under weather perturbations. The top row shows clean images with correct labels, while the bottom row shows weather-corrupted adversarial samples with misclassifications.
• Figure 4: (a) Examples of adversarial sample attacks on the image captioning task; (b) Examples of adversarial sample attacks on the VQA task.
• Figure 5: Stage 1 hyperparameter ablation results.