TACO: Adversarial Camouflage Optimization on Trucks to Fool Object Detectors

TL;DR

TACO addresses the vulnerability of modern object detectors to physical adversarial attacks on 3D vehicles by leveraging Unreal Engine 5 and a differentiable rendering pipeline to optimize adversarial textures. The framework combines a neural renderer with a Photorealistic Rendering Network, IoP-based filtering, and a Convolutional Smooth Loss to produce visually plausible yet highly effective camouflage that deceives YOLOv8 and transfers to other detectors. Key contributions include the first UE5-based differentiable adversarial pipeline for vehicle camouflage, IoP-based bounding-box filtering, and the Convolutional Smooth Loss, enabling robust, transferable attacks with high visual fidelity. The results show near-zero AP@0.5 and ADR on unseen data and demonstrate transferability across multiple detector families, highlighting significant implications for the security of autonomous systems and the need for robust defenses.

Abstract

Adversarial attacks threaten the reliability of machine learning models in critical applications like autonomous vehicles and defense systems. As object detectors become more robust with models like YOLOv8, developing effective adversarial methodologies is increasingly challenging. We present Truck Adversarial Camouflage Optimization (TACO), a novel framework that generates adversarial camouflage patterns on 3D vehicle models to deceive state-of-the-art object detectors. Adopting Unreal Engine 5, TACO integrates differentiable rendering with a Photorealistic Rendering Network to optimize adversarial textures targeted at YOLOv8. To ensure the generated textures are both effective in deceiving detectors and visually plausible, we introduce the Convolutional Smooth Loss function, a generalized smooth loss function. Experimental evaluations demonstrate that TACO significantly degrades YOLOv8's detection performance, achieving an AP@0.5 of 0.0099 on unseen test data. Furthermore, these adversarial patterns exhibit strong transferability to other object detection models such as Faster R-CNN and earlier YOLO versions.

Figures (12)

• Figure S1: Data flow and training process of our neural renderer, composed of two elements: a differential renderer and the Photorealistic Rendering Network (PRN). The error images (absolute difference and mean squared error, magnified) are marked with an asterisk (*) to indicate that they are included for illustration purposes only.
• Figure S2: Architecture of the Photorealistic Rendering Network (PRN) based on a U-Net. The contracting path extracts features, while the expansive path reconstructs the photorealistic image. CBAM modules are used in the contracting path for attention cbam.
• Figure S3: Comparison of shadow rendering quality with and without the gray textured truck input. The figure shows, from left to right: (1) applied texture, (2) ground truth rendering from UE5, (3) output of the neural renderer excluding $X_{gray}$, and (4) output of the neural renderer including the $X_{gray}$. Shadows cast on the truck are poorly rendered without $X_{gray}$, but are accurately captured when it is included.
• Figure S4: Texture optimization framework.
• Figure S5: Comparison of IoU-based and IoP-based bounding box filtering for the class loss. Top row: IoU-based filtering results in false-positive detections on the truck surface. Bottom row: IoP-based filtering suppresses these false positives. Each column shows a different viewpoint of the same optimized texture.