Embodied Red Teaming for Auditing Robotic Foundation Models

TL;DR

This work addresses safety and generalization gaps in language-conditioned robotic systems by introducing Embodied Red Teaming (ERT), an automated evaluation framework that grounds adversarial instructions in the robot’s environment using Vision-Language Models. ERT generates diverse, feasible yet challenging instructions and iteratively refines them through robot-execution feedback, balancing task feasibility with instruction diversity via Best-of-$M$ sampling and CLIP-based embeddings. Empirical results across CALVIN, RLBench, and large VLA models (OpenVLA) show that state-of-the-art policies that excel on traditional benchmarks fail or behave unsafely on ERT-generated instructions, revealing systematic generalization and safety gaps and indicating cross-model vulnerabilities. The findings underscore the need for safety-aligned evaluation and training in grounding language for robotics, with broad practical impact for deploying language-conditioned robots in real-world settings, and the work provides a path toward scalable, environment-aware red-teaming for foundational robotic systems. $R(\pi,\boldsymbol{c})$ is the task-success indicator and $\text{Div}(\cdot)$ the diversity term, jointly optimized in the ERT framework, illustrating the balance between performance and coverage in realistic instruction distributions. $N$ denotes the number of instructions per task, and $k$ the refinement iteration, making the approach amenable to iterative improvement and reproducibility. These insights advance robust, safe, and generalizable language-conditioned robotics toward real-world deployment.

Abstract

Language-conditioned robot models have the potential to enable robots to perform a wide range of tasks based on natural language instructions. However, assessing their safety and effectiveness remains challenging because it is difficult to test all the different ways a single task can be phrased. Current benchmarks have two key limitations: they rely on a limited set of human-generated instructions, missing many challenging cases, and focus only on task performance without assessing safety, such as avoiding damage. To address these gaps, we introduce Embodied Red Teaming (ERT), a new evaluation method that generates diverse and challenging instructions to test these models. ERT uses automated red teaming techniques with Vision Language Models (VLMs) to create contextually grounded, difficult instructions. Experimental results show that state-of-the-art language-conditioned robot models fail or behave unsafely on ERT-generated instructions, underscoring the shortcomings of current benchmarks in evaluating real-world performance and safety. Code and videos are available at: https://s-karnik.github.io/embodied-red-team-project-page.

Figures (5)

• Figure 1: (a) Users may express intentions in various ways, but language-conditioned robots often succeed with some instructions and fail with others, highlighting their limited ability to generalize. (b) Each task can be described by multiple instructions. Embodied Red Teaming (ERT) uses a task description and environmental observations (e.g., camera images) to generate instructions that are likely to cause the robot to fail. By incorporating feedback from the robot's execution results, ERT refines the instruction generation process to increase the probability of failure.
• Figure 2: Example environments from CALVIN and RLBench.
• Figure 3: The average success rates of the GR-1 and 3D-Diffuser models were evaluated on two instruction sets: (a) CALVIN and (b) RLBench. Both models performed nearly optimally on existing instructions in both environments but showed significant performance drops on instructions generated by ERT. This indicates that the current language-conditioned robot models are overfitting to narrow instruction sets and fail to generalize, despite using large-scale pre-trained language embeddings.
• Figure 4: Instruction Diversity. BLEU diversity captures variations in text form, while CLIP and BERT diversity measure semantic differences. Since all instructions describe the same task, semantic diversity is similar across methods. Regardless of the number of refinement $k$, ERT achieves higher BLEU diversity, indicating it generates more varied phrasing for the same task compared to other methods. See Section \ref{['fig:perf']}.
• Figure 5: Robots (3D-Diffuser ke20243d in this case) may follow unsafe instructions, as seen in Figures \ref{['subfig:unsafe_safety_1']} and \ref{['subfig:unsafe_safety_2']}, or exhibit unsafe behavior even with neutral instructions, shown in Figures \ref{['subfig:neutral_safety_1']} and \ref{['subfig:neutral_safety_2']}. This highlights the need for alignment training to ensure robots reject harmful tasks. Additionally, a potential vulnerability exists in that malicious users could exploit seemingly neutral instructions to cause harm, posing a significant safety risk.