CoDefend: Cross-Modal Collaborative Defense via Diffusion Purification and Prompt Optimization

TL;DR

CoDefend tackles multimodal adversarial threats to vision–language models by integrating a supervised diffusion‑based image purifier with cross‑modal prompt optimization. The purifier is trained on paired adversarial–clean images in latent space to perform directional, task‑oriented denoising while keeping the vision–language backbone fixed, and a LoRA‑based prefix generator adds defense‑driven text prompts to user queries. Empirical results on image captioning and VQA show near‑complete mitigation of adversarial attacks under in‑distribution settings and strong transferability to unseen attacks, with limited degradation on clean inputs. The approach offers a practical, black‑box defense with competitive robustness and transferability, though inference latency from diffusion and prefix steps motivates efficiency improvements for real‑time deployment.

Abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable success in tasks such as image captioning, visual question answering, and cross-modal reasoning by integrating visual and textual modalities. However, their multimodal nature also exposes them to adversarial threats, where attackers can perturb either modality or both jointly to induce harmful, misleading, or policy violating outputs. Existing defense strategies, such as adversarial training and input purification, face notable limitations: adversarial training typically improves robustness only against known attacks while incurring high computational costs, whereas conventional purification approaches often suffer from degraded image quality and insufficient generalization to complex multimodal tasks. In this work, we focus on defending the visual modality, which frequently serves as the primary entry point for adversarial manipulation. We propose a supervised diffusion based denoising framework that leverages paired adversarial clean image datasets to fine-tune diffusion models with directional, task specific guidance. Unlike prior unsupervised purification methods such as DiffPure, our approach achieves higher quality reconstructions while significantly improving defense robustness in multimodal tasks. Furthermore, we incorporate prompt optimization as a complementary defense mechanism, enhancing resistance against diverse and unseen attack strategies. Extensive experiments on image captioning and visual question answering demonstrate that our method not only substantially improves robustness but also exhibits strong transferability to unknown adversarial attacks. These results highlight the effectiveness of supervised diffusion based denoising for multimodal defense, paving the way for more reliable and secure deployment of MLLMs in real world applications.

Figures (2)

• Figure 1: An overview of CoDefend.
• Figure 2: An assessment of feature-level purification efficacy of our approach was conducted on 121 adversarial samples generated through the BARD attack dong2023robustgooglesbardadversarial, utilizing the visual encoder of BLIP-2.