TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Models

TL;DR

This work tackles the vulnerability of vision-language models like CLIP to adversarial perturbations by introducing Test-Time Padding (TTP), a lightweight detect-then-adapt defense. TTP detects adversarial inputs via a cosine similarity shift between embeddings before and after fixed padding, using a universal threshold, and leaves clean inputs unchanged. For detected adversaries, it applies trainable padding optimized by entropy minimization over augmented views and employs a similarity-aware ensemble to stabilize predictions, achieving strong robustness without sacrificing clean accuracy. Across multiple CLIP backbones and eight fine-grained datasets, TTP outperforms state-of-the-art test-time defenses and remains compatible with existing test-time adaptation techniques, offering a practical and generalizable defense for vision-language systems.

Abstract

Vision-Language Models (VLMs), such as CLIP, have achieved impressive zero-shot recognition performance but remain highly susceptible to adversarial perturbations, posing significant risks in safety-critical scenarios. Previous training-time defenses rely on adversarial fine-tuning, which requires labeled data and costly retraining, while existing test-time strategies fail to reliably distinguish between clean and adversarial inputs, thereby preventing both adversarial robustness and clean accuracy from reaching their optimum. To address these limitations, we propose Test-Time Padding (TTP), a lightweight defense framework that performs adversarial detection followed by targeted adaptation at inference. TTP identifies adversarial inputs via the cosine similarity shift between CLIP feature embeddings computed before and after spatial padding, yielding a universal threshold for reliable detection across architectures and datasets. For detected adversarial cases, TTP employs trainable padding to restore disrupted attention patterns, coupled with a similarity-aware ensemble strategy for a more robust final prediction. For clean inputs, TTP leaves them unchanged by default or optionally integrates existing test-time adaptation techniques for further accuracy gains. Comprehensive experiments on diverse CLIP backbones and fine-grained benchmarks show that TTP consistently surpasses state-of-the-art test-time defenses, delivering substantial improvements in adversarial robustness without compromising clean accuracy. The code for this paper will be released soon.

Figures (5)

• Figure 1: Visualization of attention maps for clean sample, adversarially perturbed sample, randomly padded sample, and samples processed with trainable test-time padding. The adversarial attack causes a noticeable shift in attention, leading to incorrect predictions. Applying random padding helps restore the original attention focus, while trainable padding further refines the attention to the correct regions and suppresses noise, resulting in more accurate predictions.
• Figure 2: Detection accuracy of TTP (ours) and TTC xing2025clip across fine-grained classification datasets under three CLIP backbones (ViT-B/32, ViT-B/16, and ViT-L/14). All experiments are performed under the same attack strength of $\epsilon=4.0$. TTC adopts its default $L_2$-distance threshold $\tau=0.2$, but exhibits pronounced fluctuations in detection performance across both datasets and backbones, indicating its sensitivity to domain and model variations. In contrast, our TTP employs a unified cosine similarity threshold $\tau=0.8$, yet maintains consistently superior detection accuracy across all settings. This demonstrates that TTP achieves not only outstanding adversarial recognition capability but also remarkable cross-dataset and cross-backbone stability, effectively mitigating the instability observed in TTC under identical conditions.
• Figure 3: Overview of the proposed Test-Time Padding (TTP) pipeline. Given an input sample, CLIP image encoder features are extracted before and after applying padding. Their cosine similarity difference is compared with a universal threshold to distinguish clean versus adversarial inputs. Clean samples are directly recognized without adaptation. For adversarial examples, trainable test-time padding is activated to optimize padding parameters by entropy minimization using augmented views with low entropy. A similarity-aware ensemble then aggregates predictions across selected high-confidence views, ensuring a more reliable final prediction. Together, TTP enables accurate adversarial detection and adaptation-driven robustness improvement.
• Figure 4: Impact of padding size on adversarial detection and robust adaptation. ViT-B/32 is used as the CLIP backbone. The figure comprises three subplots: (a) average cosine similarities on fine-grained classification datasets of CLIP embeddings before and after padding across varying padding sizes, (b) detection accuracy for both adversarial and clean inputs, and (c) adversarial accuracy on the DTD dataset.
• Figure 5: Detection accuracy of TTP with CLIP-ViT-B/32 ($\epsilon=4.0$) under varying threshold values.