ReGen: Generative Robot Simulation via Inverse Design

TL;DR

ReGen addresses the challenge of scalable, diverse robot simulation by introducing an inverse-design workflow that starts from a robot behavior and its description and builds an LLM-guided causal graph of plausible environments. This graph is transformed into a symbolic program grounding a simulator via a finite-state machine, enabling controllable, counterfactual scenario generation, ego-agent augmentation, and multimodal sensing reasoning. Empirical results in autonomous driving and manipulation show increased environmental diversity, effective corner-case generation, and the production of more complex vision-language-action datasets than prior works, with analyses of uncertainty and failure modes. The approach offers a scalable pathway to validate policies and augment data, potentially improving generalization and robustness in scalable robot learning.

Abstract

Simulation plays a key role in scaling robot learning and validating policies, but constructing simulations remains a labor-intensive process. This paper introduces ReGen, a generative simulation framework that automates simulation design via inverse design. Given a robot's behavior -- such as a motion trajectory or an objective function -- and its textual description, ReGen infers plausible scenarios and environments that could have caused the behavior. ReGen leverages large language models to synthesize scenarios by expanding a directed graph that encodes cause-and-effect relationships, relevant entities, and their properties. This structured graph is then translated into a symbolic program, which configures and executes a robot simulation environment. Our framework supports (i) augmenting simulations based on ego-agent behaviors, (ii) controllable, counterfactual scenario generation, (iii) reasoning about agent cognition and mental states, and (iv) reasoning with distinct sensing modalities, such as braking due to faulty GPS signals. We demonstrate ReGen in autonomous driving and robot manipulation tasks, generating more diverse, complex simulated environments compared to existing simulations with high success rates, and enabling controllable generation for corner cases. This approach enhances the validation of robot policies and supports data or simulation augmentation, advancing scalable robot learning for improved generalization and robustness. We provide code and example videos at: https://regen-sim.github.io/

Figures (8)

• Figure 1: Given a robot behavior (such as a trajectory or an underlying objective function) and its textual description, ReGen generates simulated environments that could have caused the behavior.
• Figure 2: ReGen for driving. Given a behavior such as "changing lanes," our method can generate diverse simulated environments in which the behavior could have occurred, such as changing lanes to "yield for an emergency vehicle", "overtaking a truck", "merging into an open lane", or "avoiding debris." The bottom row illustrates additional cases: "ego-vehicle slows down $\leftarrow$ avoids collision" and "ego-vehicle stops $\leftarrow$ halts for an opening car door."
• Figure 4: Emergent capabilities of ReGen. (1) Our method can simulate scenarios that capture the mental states or decision-making process of actors---for example, a distracted driver failing to move despite a green light. (2) Our method can model reasoning across different sensing modalities, such as simulating a scenario with GPS jamming with noisy GNSS measurements in CARLA. (3) Our method enables counterfactual scenario generation by perturbing the constructed graph with "what-if" questions---for example, changing from "the front car stops with brake lights" to "the front car with broken brake light stops and thus brake light being off."
• Figure 5: Diversity of Corner Cases. Compared to ChatScene zhang2024chatsceneknowledgeenabledsafetycriticalscenario, our method generates more diverse corner cases via reasoning about different causes.
• Figure 6: Controllability