ReGentS: Real-World Safety-Critical Driving Scenario Generation Made Stable

TL;DR

Safety-critical driving scenarios are scarce in real-world data, limiting robust planner training. ReGentS stabilizes gradient-based trajectory optimization by applying constraints and heuristics within a differentiable, large-scale simulator to generate realistic ego–adversary collisions with up to 32 agents. The approach analyzes optimization biases, introduces remedies such as Stop Front Divergence and Exclude Rear-End/Static Adversaries, and demonstrates improved collision-generation effectiveness on the Waymo Open Motion Dataset using Waymax. Overall, ReGentS yields more diverse, plausible corner cases than prior KING-based methods, enabling scalable, real-data-driven safety evaluation for autonomous driving systems.

Abstract

Machine learning based autonomous driving systems often face challenges with safety-critical scenarios that are rare in real-world data, hindering their large-scale deployment. While increasing real-world training data coverage could address this issue, it is costly and dangerous. This work explores generating safety-critical driving scenarios by modifying complex real-world regular scenarios through trajectory optimization. We propose ReGentS, which stabilizes generated trajectories and introduces heuristics to avoid obvious collisions and optimization problems. Our approach addresses unrealistic diverging trajectories and unavoidable collision scenarios that are not useful for training robust planner. We also extend the scenario generation framework to handle real-world data with up to 32 agents. Additionally, by using a differentiable simulator, our approach simplifies gradient descent-based optimization involving a simulator, paving the way for future advancements. The code is available at https://github.com/valeoai/ReGentS.

Figures (6)

• Figure 1: Importance of Constraints in Ego-Adversary Collision Scenarios. The figures depict the final state with past trajectories shown as dotted lines. The original collision-free scenario \ref{['fig:teaser-original']} is modified by an unconstrained method hanselmann2022king\ref{['fig:teaser-king']} and our ReGentS \ref{['fig:teaser-ours']} with proposed constraints. The ego vehicle is at the center, and the adversary is in front. In \ref{['fig:teaser-king']}, the unconstrained adversary takes an unrealistic swinging turn (first to the left then to the right), leading to a partially diverging trajectory that does not reflect plausible real-world driving behavior. In contrast, our ReGentS \ref{['fig:teaser-ours']} ensures the adversary decelerates smoothly while gradually approaching the ego, maintaining a realistic and natural trajectory.
• Figure 2: Forward Model Assumption of ReGentS Through a Differentiable Simulator, accompanied by the agent's structure for state transition from $s_k$ to $s_{k+1}$. The ego agent ($i=0$) generates an observation $o_k$ by rasterizing the scenario onto a uniform grid. The planner uses this observation and a target point $x_\text{target}$ to predict the ego agent's action $a_k^{(0)}$ with a controller, which is fed into the kinematic model $f$. For background agents ($i>0$), the model $f$ estimates their actions $a^{(i)}$ and reconstructs their logged trajectory $\tilde{q}^{(i)}$. Arrows represent the computational graph which enables the backward autodiff.
• Figure 3: Illustration of Representative Issues with Unconstrained KING Method hanselmann2022king. The final state of each scenario is shown along with the trajectory history in dotted line. The ego agent (blue) is at the center, and the adversary agent (green) represents the last optimized adversary.
• Figure 4: Negative Gradient of the Distance Between the Ego Agent ($\rightarrow$) and an Adversary ($\rightarrow$) w.r.t. the Adversary's Actions. In \ref{['fig:schema-steer']}, two interesting zones are highlighted. Diverging Front Adversary zone (red): Adversaries could pursue a diverging trajectory w.r.t. the ego agent. The negative gradient points to the diverging direction. The angle this zone is determined by the yaw of the adversary. Converging Rear-end Adversary zone (orange): Adversaries are likely to rear-end the ego agent. After one gradient descent update from \ref{['fig:schema-steer']} to \ref{['fig:schema-steer-after']}, the negative gradient in the zoomed area intensifies, favoring the divergence of the adversary.
• Figure 5: Illustration of how to determine if an adversary is in the red zone, with the red zone calculation shown. The ego agent is indicated by ($\rightarrow$) and the adversary by ($\rightarrow$).