Predictive Red Teaming: Breaking Policies Without Breaking Robots

TL;DR

The paper addresses the challenge of predicting how visuomotor policies fail under unseen environmental factors without hardware testing. It formalizes predictive red teaming and introduces RoboART, a pipeline that edits nominal observations via generative image editing and then uses a policy-specific anomaly detector to forecast performance degradation, allowing rank-based vulnerability assessment and absolute performance estimation. Empirical results across 12 off-nominal factors and 500+ hardware trials show RoboART achieving high predictive accuracy (average error $< 0.19$) and enabling targeted data collection that boosts performance by $2$–$7\times$ and improves cross-domain generalization by $2$–$5\times$. The work highlights the practical impact of anticipating deployments’ limits, guiding data collection, policy comparison, and safer, more robust real-world robot operation. It also discusses limitations such as edit-to-real gaps and single-time-step anomaly estimates, pointing to future work in 3D scene editing and iterative exploration of environmental factor spaces.

Abstract

Visuomotor policies trained via imitation learning are capable of performing challenging manipulation tasks, but are often extremely brittle to lighting, visual distractors, and object locations. These vulnerabilities can depend unpredictably on the specifics of training, and are challenging to expose without time-consuming and expensive hardware evaluations. We propose the problem of predictive red teaming: discovering vulnerabilities of a policy with respect to environmental factors, and predicting the corresponding performance degradation without hardware evaluations in off-nominal scenarios. In order to achieve this, we develop RoboART: an automated red teaming (ART) pipeline that (1) modifies nominal observations using generative image editing to vary different environmental factors, and (2) predicts performance under each variation using a policy-specific anomaly detector executed on edited observations. Experiments across 500+ hardware trials in twelve off-nominal conditions for visuomotor diffusion policies demonstrate that RoboART predicts performance degradation with high accuracy (less than 0.19 average difference between predicted and real success rates). We also demonstrate how predictive red teaming enables targeted data collection: fine-tuning with data collected under conditions predicted to be adverse boosts baseline performance by 2-7x.

Figures (9)

• Figure 1: We propose predictive red teaming: discovering vulnerabilities of a policy with respect to environmental factors and predicting the corresponding performance degradation without hardware evaluations in off-nominal scenarios. Our approach modifies nominal observations using generative image editing to reflect changes in environmental factors (e.g., background, lighting, injecting humans and other distractors), and predicts the resulting performance degradation via anomaly detection.
• Figure 2: We evaluate RoboART's predictions using 500+ hardware trials in twelve off-nominal conditions.
• Figure 3: Examples of adding a novel object to the visual scene via generative image editing. Original (top) and edited (bottom) observations from both the robot's overhead and wrist cameras are shown. State-of-the-art generative image editing tools render the desired object in a realistic manner that maintains per-view global consistency in lighting, shadows, and overall composition.
• Figure 4: A vision-language model ensures that the edited image reflects the desired change.
• Figure 5: Evaluating predictions from RoboART for $\pi_\text{hyb}$ (which combines trajectory optimization with diffusion) in the top panel and $\pi_\text{hyb}$ (vanilla diffusion policy) in the bottom panel. Left: Comparison of true (estimated) rankings of different environmental factors by performance degradation with predictions made by RoboART. Right: Comparison of true (estimated) success rates with predictions from RoboART.