Adversarial Robustness for Unified Multi-Modal Encoders via Efficient Calibration

TL;DR

This work investigates adversarial robustness of unified multi-modal encoders (Bind-style) across six modalities and three architectures, revealing that small perturbations can substantially disrupt cross-modal embeddings under budgets $\epsilon \in \{2,4,8\}/255$. It introduces CALMARS, a two-stage defense that preserves frozen encoders and semantic centers while training lightweight modality-specific projection heads, and adds CrossMaxim, an untargeted retrieval attack that probes the entire similarity space. CALMARS achieves state-of-the-art robustness–accuracy trade-offs, improving adversarial robustness by up to 47.3% at $\epsilon=4/255$ and often maintaining or enhancing clean zero-shot and retrieval performance, with trainable parameters falling under 1% of the encoder. The approach demonstrates strong cross-modal and cross-VLM transfer robustness, validates the importance of alignment and stability in multi-modal spaces, and provides a practical, plug-in defense for safety-critical applications.

Abstract

Recent unified multi-modal encoders align a wide range of modalities into a shared representation space, enabling diverse cross-modal tasks. Despite their impressive capabilities, the robustness of these models under adversarial perturbations remains underexplored, which is a critical concern for safety-sensitive applications. In this work, we present the first comprehensive study of adversarial vulnerability in unified multi-modal encoders. We find that even mild adversarial perturbations lead to substantial performance drops across all modalities. Non-visual inputs, such as audio and point clouds, are especially fragile, while visual inputs like images and videos also degrade significantly. To address this, we propose an efficient adversarial calibration framework that improves robustness across modalities without modifying pretrained encoders or semantic centers, ensuring compatibility with existing foundation models. Our method introduces modality-specific projection heads trained solely on adversarial examples, while keeping the backbone and embeddings frozen. We explore three training objectives: fixed-center cross-entropy, clean-to-adversarial L2 alignment, and clean-adversarial InfoNCE, and we introduce a regularization strategy to ensure modality-consistent alignment under attack. Experiments on six modalities and three Bind-style models show that our method improves adversarial robustness by up to 47.3 percent at epsilon = 4/255, while preserving or even improving clean zero-shot and retrieval performance with less than 1 percent trainable parameters.

Figures (2)

• Figure 1: Overview of the CALMARS framework. Stage 1 CALM performs clean feature distillation and embedding calibration with a frozen encoder $\varphi(X)$ and a lightweight projection head $h_\theta$. Stage 2 MARS refines both clean and adversarial inputs $(X, X_{\text{adv}})$ through geometric consistency, feature regularization, and robust calibration. The projection head $h_\theta$ has less than 1% of the encoder’s parameters.
• Figure 2: Comparison of clean-alignment methods across eight datasets on UniBind. Each line represents a distillation-based baseline (Bind, CLIP-KD, DCLIP) or our CALM method.