Adversarial Patch Attacks on Vision-Based Cargo Occupancy Estimation via Differentiable 3D Simulation

TL;DR

This paper addresses the vulnerability of vision-based cargo occupancy estimation to physically realizable adversarial patches. It presents a differentiable 3D simulation framework using Mitsuba 3 to optimize patches placed on trailer interiors, comparing 3D scene-space optimization against a 2D image-space baseline. The study finds that 3D-optimized patches achieve high attack success, particularly in Denial-of-Service cases where ASR reaches $84.94\%$, while Concealment remains more challenging at $30.32\%$. The results underscore the importance of physical realism in evaluating adversarial robustness and motivate defenses such as adversarial training and sensor fusion to strengthen logistics pipelines against such threats.

Abstract

Computer vision systems are increasingly adopted in modern logistics operations, including the estimation of trailer occupancy for planning, routing, and billing. Although effective, such systems may be vulnerable to physical adversarial attacks, particularly adversarial patches that can be printed and placed on interior surfaces. In this work, we study the feasibility of such attacks on a convolutional cargo-occupancy classifier using fully simulated 3D environments. Using Mitsuba 3 for differentiable rendering, we optimize patch textures across variations in geometry, lighting, and viewpoint, and compare their effectiveness to a 2D compositing baseline. Our experiments demonstrate that 3D-optimized patches achieve high attack success rates, especially in a denial-of-service scenario (empty to full), where success reaches 84.94 percent. Concealment attacks (full to empty) prove more challenging but still reach 30.32 percent. We analyze the factors influencing attack success, discuss implications for the security of automated logistics pipelines, and highlight directions for strengthening physical robustness. To our knowledge, this is the first study to investigate adversarial patch attacks for cargo-occupancy estimation in physically realistic, fully simulated 3D scenes.

Figures (5)

• Figure 1: Illustrative example of a successful adversarial patch attack in our simulated cargo trailer environment. A learned patch placed inside the trailer changes the prediction of a high-accuracy occupancy classifier.
• Figure 2: Example of a learned patch rendered under different lighting conditions. Differentiable rendering through Mitsuba 3 allows us to optimize patch textures across illumination variations, improving physical plausibility.
• Figure 3: Per-class performance of the victim model on clean test images. Accuracy, precision, recall, and F1-score are all above 0.99 for each occupancy class.
• Figure 4: Examples of successful Denial-of-Service attacks (empty $\to$ full) using 3D-optimized patches. The patches are integrated into the scene with realistic lighting and occlusion, leading the classifier to predict a full trailer.
• Figure 5: Examples of successful Concealment attacks (full $\to$ empty) in the 3D-rendered environment. Patches placed directly on cargo surfaces interfere with the visual features used by the classifier, causing it to misclassify full trailers as empty.