Towards Transferable Targeted 3D Adversarial Attack in the Physical World

TL;DR

TT3D tackles transferable targeted 3D adversarial attacks in the physical world by reconstructing a textured 3D mesh from multi-view data and optimizing it in grid-based NeRF space. It achieves this through dual optimization over the appearance grid and its MLP while optionally perturbing vertex geometry, aiming to minimize the target misclassification loss plus a naturalness regularizer under an EOT framework, with optimization variables $\Theta_{G_{tex}}$, $\Theta_{M_{tex}}$, and $\mathcal{V}$. Regularization combines appearance and geometric constraints via $R = \lambda_1 R_{rgb} + \lambda_2 R_{cd} + \lambda_3 R_{lap} + \lambda_4 R_{edge}$, and physical robustness is enforced by $\hat I_{\boldsymbol{v}}(\mathcal{M}_{adv}) = t(\mathcal{S}(\mathcal{V}^*, \mathcal{T}^*, \mathcal{F}, \rho(\boldsymbol{v})))$ with transformations $t$ and $\rho$. Experiments demonstrate strong cross-model transferability across multiple backbones and renders, as well as successful physical-world attacks using 3D printing, highlighting TT3D's practical potential. Overall, TT3D expands the feasibility of robust, transferable targeted 3D adversarial attacks in real-world settings.

Abstract

Compared with transferable untargeted attacks, transferable targeted adversarial attacks could specify the misclassification categories of adversarial samples, posing a greater threat to security-critical tasks. In the meanwhile, 3D adversarial samples, due to their potential of multi-view robustness, can more comprehensively identify weaknesses in existing deep learning systems, possessing great application value. However, the field of transferable targeted 3D adversarial attacks remains vacant. The goal of this work is to develop a more effective technique that could generate transferable targeted 3D adversarial examples, filling the gap in this field. To achieve this goal, we design a novel framework named TT3D that could rapidly reconstruct from few multi-view images into Transferable Targeted 3D textured meshes. While existing mesh-based texture optimization methods compute gradients in the high-dimensional mesh space and easily fall into local optima, leading to unsatisfactory transferability and distinct distortions, TT3D innovatively performs dual optimization towards both feature grid and Multi-layer Perceptron (MLP) parameters in the grid-based NeRF space, which significantly enhances black-box transferability while enjoying naturalness. Experimental results show that TT3D not only exhibits superior cross-model transferability but also maintains considerable adaptability across different renders and vision tasks. More importantly, we produce 3D adversarial examples with 3D printing techniques in the real world and verify their robust performance under various scenarios.

Figures (11)

• Figure 1: A comparison of transferable targeted attack performance in the 3D domain between our TT3D and the enhanced version of the typical mesh-based optimization method xiao2019meshadv, as detailed in \ref{['basic']}. The surrogate model is ResNet-101 He_2016_CVPR and we can see TT3D shows remarkable transferability.
• Figure 2: An overview of our TT3D framework. We first utilize 3D multi-view reconstruction technology, i.e., grid-based NeRF with marching cubes techniques to obtain the initial clean 3D mesh. Then, we perform adversarial fine-tuning in the textual parameter space of grid-based NeRF instead of directly altering the texture $\mathcal{T}$, supplemented with geometric perturbations at vertex positions $\mathcal{V}$. To ensure the naturalness simultaneously, we add constraints to the distance between the 3D adversarial samples and the initial ones in terms of both texture and geometric structure when performing optimization .
• Figure 3: Visual examples of original objects, our TT3D and mesh-based optimization methods under random viewpoints.
• Figure 4: Comparison between rendering results of different renders, including differential rendering library Nvdiffast, commercial software Meshlab, and Blender.
• Figure 5: The prediction examples of our 3D adversarial examples targeting zero-shot detection and image caption tasks. Green and red text represent the clean label and the target label, respectively.