AdvDreamer Unveils: Are Vision-Language Models Truly Ready for Real-World 3D Variations?

TL;DR

Vision-Language Models exhibit notable robustness gaps to real-world 3D variations, motivating AdvDreamer to systematically probe these weaknesses. AdvDreamer combines a zero-shot Monocular Pose Manipulation pipeline, a Naturalness Reward Model to preserve visual realism, and an Inverse Semantic Probability loss to ensure cross-model transferability, optimized via CMA-ES to generate aggressive Adv-3DT samples from single-view images. The authors introduce MM3DTBench, a 3D-variation robustness benchmark for VLMs, and demonstrate across zero-shot classification, image captioning, and VQA that real-world 3D changes can cause severe performance losses, with substantial cross-model transferability and partial recoveries when naturalness constraints are applied. These findings underscore the urgent need to strengthen 3D perception and alignment in VLMs for safe, real-world deployment and provide a practical benchmark to drive future improvements.

Abstract

Vision Language Models (VLMs) have exhibited remarkable generalization capabilities, yet their robustness in dynamic real-world scenarios remains largely unexplored. To systematically evaluate VLMs' robustness to real-world 3D variations, we propose AdvDreamer, the first framework capable of generating physically reproducible Adversarial 3D Transformation (Adv-3DT) samples from single-view observations. In AdvDreamer, we integrate three key innovations: Firstly, to characterize real-world 3D variations with limited prior knowledge precisely, we design a zero-shot Monocular Pose Manipulation pipeline built upon generative 3D priors. Secondly, to ensure the visual quality of worst-case Adv-3DT samples, we propose a Naturalness Reward Model that provides continuous naturalness regularization during adversarial optimization, effectively preventing convergence to hallucinated or unnatural elements. Thirdly, to enable systematic evaluation across diverse VLM architectures and visual-language tasks, we introduce the Inverse Semantic Probability loss as the adversarial optimization objective, which solely operates in the fundamental visual-textual alignment space. Based on the captured Adv-3DT samples with high aggressiveness and transferability, we establish MM3DTBench, the first VQA benchmark dataset tailored to evaluate VLM robustness under challenging 3D variations. Extensive evaluations of representative VLMs with varying architectures reveal that real-world 3D variations can pose severe threats to model performance across various tasks.

Figures (7)

• Figure 1: The 3D Variation Vulnerabilities in VLMs. (A) The proposed AdvDreamer allow the capture of adversarial 3D transformation (Adv-3DT) samples from single natural images, threatening various VLMs. (B) These Adv-3DT samples can be successfully reproduced in the physical world and (C) retain their aggressiveness under background variations. (D) We also explain the GPT-Score metric used in the experiments. In each VLM response, we highlight the correct, significant error, and less precise parts using different colors.
• Figure 2: Comparison of paradigms to characterize Adversarial 3D Transformation (Adv-3DT) samples.
• Figure 3: Overview of the proposed AdvDreamer framework. To capture worst-case 3D variations in the real world, we treat rigid 3D transformations as adversarial attacks, optimizing the distribution of transformation parameters through a query-based approach. In AdvDreamer, we introduce the Monocular Pose Manipulation pipeline to perform specified 3D transformations on single-view images and guide the optimization process using the proposed Naturalness Reward Model and Inverse Semantic Probability loss.
• Figure 4: Architecture of NRM and computational cost and accuracy of naturalness assessment across Human, GPT, and NRM.
• Figure 5: Visualization of Adv-3DT samples w/ and w/o NRM.