Flexible Physical Camouflage Generation Based on a Differential Approach

TL;DR

The paper tackles robust physical adversarial camouflage for 3D objects by introducing Flexible Physically-camouflage Attack (FPA), which combines a differentiable 3D renderer with diffusion-model-based texture generation and a dual loss design. It optimizes UV-mapped textures to produce adversarial samples from multiple viewpoints while accounting for lighting and material variations, guided by the objective $L_{ADV}$ and the concealment loss $L_{CR}$. Key contributions include UV-map-based diffusion texture generation, a differentiable rendering pipeline that models real-world factors, and a loss framework that balances attack efficacy with concealment, demonstrated across diverse detectors and tasks. Results show high attack success and strong transferability in simulation and the physical world, including under occlusion and varying illumination, indicating practical potential and highlighting limitations in handling non-rigid deformations.

Abstract

This study introduces a novel approach to neural rendering, specifically tailored for adversarial camouflage, within an extensive 3D rendering framework. Our method, named FPA, goes beyond traditional techniques by faithfully simulating lighting conditions and material variations, ensuring a nuanced and realistic representation of textures on a 3D target. To achieve this, we employ a generative approach that learns adversarial patterns from a diffusion model. This involves incorporating a specially designed adversarial loss and covert constraint loss to guarantee the adversarial and covert nature of the camouflage in the physical world. Furthermore, we showcase the effectiveness of the proposed camouflage in sticker mode, demonstrating its ability to cover the target without compromising adversarial information. Through empirical and physical experiments, FPA exhibits strong performance in terms of attack success rate and transferability. Additionally, the designed sticker-mode camouflage, coupled with a concealment constraint, adapts to the environment, yielding diverse styles of texture. Our findings highlight the versatility and efficacy of the FPA approach in adversarial camouflage applications.

Figures (14)

• Figure 1: Workflow of generating robust adversarial textures in FPA framework.
• Figure 2: (a) shows the results generated based on the diffusion model, while (b) shows the results generated based on the gradient method.
• Figure 3: Comparison of rendered images with different textures using the Neural Renderer and the Flexible Renderer.
• Figure 4: The presentation of adversarial camouflage and the prediction results under the YOLOv5 detection model.
• Figure 5: The first row shows the prediction results of original samples under various detectors, and the second row shows the prediction results of our adversarial camouflage, FPA(v5).