NutVLM: A Self-Adaptive Defense Framework against Full-Dimension Attacks for Vision Language Models in Autonomous Driving

TL;DR

The proposed NutVLM is a comprehensive self-adaptive defense framework designed to secure the entire perception-decision lifecycle of Vision Language Models, and its results validate NutVLM as a scalable security solution for intelligent transportation.

Abstract

Vision Language Models (VLMs) have advanced perception in autonomous driving (AD), but they remain vulnerable to adversarial threats. These risks range from localized physical patches to imperceptible global perturbations. Existing defense methods for VLMs remain limited and often fail to reconcile robustness with clean-sample performance. To bridge these gaps, we propose NutVLM, a comprehensive self-adaptive defense framework designed to secure the entire perception-decision lifecycle. Specifically, we first employ NutNet++ as a sentinel, which is a unified detection-purification mechanism. It identifies benign samples, local patches, and global perturbations through three-way classification. Subsequently, localized threats are purified via efficient grayscale masking, while global perturbations trigger Expert-guided Adversarial Prompt Tuning (EAPT). Instead of the costly parameter updates of full-model fine-tuning, EAPT generates "corrective driving prompts" via gradient-based latent optimization and discrete projection. These prompts refocus the VLM's attention without requiring exhaustive full-model retraining. Evaluated on the Dolphins benchmark, our NutVLM yields a 4.89% improvement in overall metrics (e.g., Accuracy, Language Score, and GPT Score). These results validate NutVLM as a scalable security solution for intelligent transportation. Our code is available at https://github.com/PXX/NutVLM.

Figures (6)

• Figure 1: Intuitive example of verifying the security of NutVLM in adversarial scenarios. Standard VLMs can be misled by adversarial examples and generate hazardous commands (marked in red), while NutVLM detects and purifies the input to yield safe navigation instructions (marked in green).
• Figure 2: Overview of the NutVLM defense framework. The architecture integrates NutNet++ as a unified detection module to identify local patches and global perturbations. Depending on the detected threat, the framework executes either pixel-level grayscale masking for patch purification or EAPT for instruction correction. This dual-branch approach ensures robust perception and safe inference for AD VLMs.
• Figure 3: Statistical distributions of key defense metrics across varying CADA attack intensities (both global and local adversarial scenarios). (a) Anomaly Magnitude $M_{anom}$, (b) Energy Entropy $H_{energy}$, (c) Local Concentration $C_{local}$, and (d) the Enhanced Concentration $C_{enh}$.
• Figure 4: Qualitative performance of NutVLM in safety-critical driving scenarios. (a) Scene description recovery on CADA-Sense, and (b) sign recognition on CADA-Obj. NutVLM effectively mitigates adversarial hallucinations (red) and restores accurate reasoning (green).
• Figure 5: Quantitative evaluation of NutNet++ performance. (a) Confusion matrices on general datasets (DAmageNet and APRICOT), and (b) Confusion matrices and ROC curves on AD-specific datasets (CADA).