On the Adversarial Robustness of 3D Large Vision-Language Models

TL;DR

The experiments reveal that 3D VLMs exhibit significant adversarial vulnerabilities under untargeted attacks, while demonstrating greater resilience against targeted attacks aimed at forcing specific harmful outputs, compared to their 2D counterparts.

Abstract

3D Vision-Language Models (VLMs), such as PointLLM and GPT4Point, have shown strong reasoning and generalization abilities in 3D understanding tasks. However, their adversarial robustness remains largely unexplored. Prior work in 2D VLMs has shown that the integration of visual inputs significantly increases vulnerability to adversarial attacks, making these models easier to manipulate into generating toxic or misleading outputs. In this paper, we investigate whether incorporating 3D vision similarly compromises the robustness of 3D VLMs. To this end, we present the first systematic study of adversarial robustness in point-based 3D VLMs. We propose two complementary attack strategies: \textit{Vision Attack}, which perturbs the visual token features produced by the 3D encoder and projector to assess the robustness of vision-language alignment; and \textit{Caption Attack}, which directly manipulates output token sequences to evaluate end-to-end system robustness. Each attack includes both untargeted and targeted variants to measure general vulnerability and susceptibility to controlled manipulation. Our experiments reveal that 3D VLMs exhibit significant adversarial vulnerabilities under untargeted attacks, while demonstrating greater resilience against targeted attacks aimed at forcing specific harmful outputs, compared to their 2D counterparts. These findings highlight the importance of improving the adversarial robustness of 3D VLMs, especially as they are deployed in safety-critical applications.

Figures (8)

• Figure 1: Generated captions on clean (left) and adversarially perturbed (right) point clouds. The figure illustrates the effects of untargeted vision attack on GPT4Point model. The output captions change significantly even under minor perturbations, revealing the adversarial fragility of current 3D VLMs.
• Figure 2: Our proposed adversarial attacks for 3D VLMs. The left panel illustrates the vision attack under both untargeted and targeted settings. This attack perturbs the high-dimensional token features extracted by the point encoder and projector. In untargeted setting, the adversarial token feature $f(x+\delta)$ is pushed away from the original feature $f(x)$, while in the targeted setting, it is aligned with the feature representation of a target point cloud $f(x_{\text{tar}})$. Cosine similarity loss guides this feature manipulation. The right panel depicts the caption attack. In the untargeted setting, the generated caption $c_{x+\delta}$ is encouraged to diverge from the ground-truth caption $c_{\text{gt}}$, whereas in the targeted setting, the model is guided to produce a specific target caption $c_{\text{tar}}$. Cross-entropy loss is used to compute the gradients that drive the generation of adversarial examples.
• Figure 3: Comparison of Our Attack Strategies on the Generative 3D Object Classification Benchmark. The plots reveal three key findings: (1) GPT4Point consistently exhibits higher vulnerability than PointLLM under both attack settings; (2) Vision attacks outperform caption attacks in the targeted setting and surpass them in the untargeted setting after a certain deformation threshold; and (3) Untargeted attacks are significantly more effective than targeted ones, underscoring the difficulty of precise adversarial control in 3D VLMs under constrained distortion.
• Figure 4: Visualization of vision attack and caption attack in untargeted setting on the ModelNet40 and Obajverse datasets. For each example, the left shows the clean point cloud with its generated captions, while the right shows the corresponding adversarial point cloud and its output captions. The perturbations applied to the point clouds are visually subtle, yet they lead to significantly altered captions.
• Figure 5: Visualization of vision attack in targeted setting on Objaverse dataset. The left and middle columns show the original and adversarial point clouds along with their generated captions, while the right column presents the target point clouds and their ground-truth descriptions.