UNDREAM: Bridging Differentiable Rendering and Photorealistic Simulation for End-to-end Adversarial Attacks

TL;DR

UNDREAM addresses the gap between non-differentiable, high-fidelity simulators and differentiable renderers to enable end-to-end adversarial optimization in realistic 3D environments. By integrating Unreal Engine with the Mitsuba differentiable renderer, it allows adversarial textures to be optimized directly within photorealistic scenes, preserving lighting and material interactions. The framework automates 3D scene transformation, supports dynamic sequences, and provides an adaptable attack pipeline, with open-source availability for reproducibility. This approach enables more faithful evaluations of attacks and defenses in applied safety-critical settings, such as autonomous systems, under varied environmental conditions.

Abstract

Deep learning models deployed in safety critical applications like autonomous driving use simulations to test their robustness against adversarial attacks in realistic conditions. However, these simulations are non-differentiable, forcing researchers to create attacks that do not integrate simulation environmental factors, reducing attack success. To address this limitation, we introduce UNDREAM, the first software framework that bridges the gap between photorealistic simulators and differentiable renderers to enable end-to-end optimization of adversarial perturbations on any 3D objects. UNDREAM enables manipulation of the environment by offering complete control over weather, lighting, backgrounds, camera angles, trajectories, and realistic human and object movements, thereby allowing the creation of diverse scenes. We showcase a wide array of distinct physically plausible adversarial objects that UNDREAM enables researchers to swiftly explore in different configurable environments. This combination of photorealistic simulation and differentiable optimization opens new avenues for advancing research of physical adversarial attacks.

Figures (7)

• Figure 1: UnDREAM is the first software framework bridging differentiable rendering and photorealistic simulation to enable end-to-end adversarial attacks. Users can create diverse environments by controlling environmental conditions, add and configure custom 3D objects and execute adversarial attacks that faithfully follow threat model.
• Figure 2: Existing attack pipelines create adversarial attacks outside of simulation without accounting for interaction with the environment, resulting in attacks that fail when the patches are integrated in the simulation.
• Figure 3: The end-to-end UnDREAM optimization pipeline. In each iteration, the adversarial texture is applied to the object in simulation, model predictions are computed, loss is calculated and the gradients are propagated back through the object in the simulation equivalent XML scene to optimize the adversarial texture.
• Figure 4: (a) UV layout and object mesh in the UV editor display in Unreal Engine 5. Each segment from the UV map corresponds to a part on the object. (b) Optimized object (sphere) shown from the side. Only the part of the texture that is visible in the camera view is optimized.
• Figure 5: UnDREAM enables optimization of 3D objects of arbitrary shapes. If the object is very large (eg: car), Unreal allows for the texture to repeat, thus covering the whole object.