Crafting Physical Adversarial Examples by Combining Differentiable and Physically Based Renders

TL;DR

The paper addresses the vulnerability of object detectors to physical adversarial camouflage for vehicles. It proposes PAV-Camou, a dual-renderer pipeline that jointly uses differentiable rendering for texture optimization and physically based rendering for photorealistic appearance, coupled with a 2D UV-mapping adjustment to minimize texture distortion and enable real-world application via printed 2D patterns. The method optimizes an adversarial texture map using a composite loss that combines detector-style objectives with a smoothing term, and fuses DR and PBR outputs through a learned mask. Experiments demonstrate strong attack performance in both digital and physical scenarios, outperforming prior methods and showing robustness to varying lighting and viewing angles, with a tractable naturalness-robustness trade-off when topology-aware constraints are applied.

Abstract

Recently we have witnessed progress in hiding road vehicles against object detectors through adversarial camouflage in the digital world. The extension of this technique to the physical world is crucial for testing the robustness of autonomous driving systems. However, existing methods do not show good performances when applied to the physical world. This is partly due to insufficient photorealism in training examples, and lack of proper physical realization methods for camouflage. To generate a robust adversarial camouflage suitable for real vehicles, we propose a novel method called PAV-Camou. We propose to adjust the mapping from the coordinates in the 2D map to those of corresponding 3D model. This process is critical for mitigating texture distortion and ensuring the camouflage's effectiveness when applied in the real world. Then we combine two renderers with different characteristics to obtain adversarial examples that are photorealistic that closely mimic real-world lighting and texture properties. The method ensures that the generated textures remain effective under diverse environmental conditions. Our adversarial camouflage can be optimized and printed in the form of 2D patterns, allowing for direct application on real vehicles. Extensive experiments demonstrated that our proposed method achieved good performance in both the digital world and the physical world.

Figures (15)

• Figure 1: Detection results of the normal car and the camouflaged car with different viewing angles. The red score (%) is the confidence of the detector Faster R-CNN. Usually attacks are regarded successful if the score is lower than 50%. (a) The normal car was detected successfully. (b) The camouflaged car significantly decreased the detection confidence in the digital world. (c) The normal car model in the physical world was detected. (d) The physical car model with our adversarial textures attached on the surface was not detected. More examples are shown in Fig. \ref{['fig:digital']} and Fig. \ref{['fig:physical']}.
• Figure 2: Overview of the optimization pipeline. DR renders the vehicle with adversarial textures, and PBR renders the vehicle with original textures. Their results are combined using masks, which are obtained by rendering the vehicle under dark light mode and binarizing the results. The textured parts of DR results are taken for gradient backpropagation, and the non-textured parts of PBR results are taken for a photorealistic appearance. In this way, adversarial textures can be tailored for attacking real vehicle detectors.
• Figure 3: Different U-V mappings of two same hemispheres and their rendering results. The first column shows two identical 3D models (hemispheres). The second column shows the U-V map, where the two trapezoids circled represent the mapping results of the same quadrilateral before and after 2D adjustments. The last column shows the rendered hemispheres after mapping. As shown in the top row, generating 2D coordinates of mapping by the direct projection can lead to texture distortion, and in the bottom row, by adjusting the coordinates, the distortion is reduced.
• Figure 4: In the same 3D car model, the U-V distortions before (a) and after (b) adjustments. Purple indicates stretching, red indicates compression, and gray indicates no distortion.
• Figure 5: The U-V map required for mask generation and its DR results under different lighting conditions. (a) The texture map $\mathbf{T}_m$ used for generating the mask. (b) The rendering result $\mathbf{I}_m$ under dark light mode. (c) The rendering result under a strong light mode. In (c), the red boxes contain certain car body areas that have similar gray values to the car doors.