Rectify, Don't Regret: Avoiding Pitfalls of Differentiable Simulation in Trajectory Prediction

Abstract

Current open-loop trajectory models struggle in real-world autonomous driving because minor initial deviations often cascade into compounding errors, pushing the agent into out-of-distribution states. While fully differentiable closed-loop simulators attempt to address this, they suffer from shortcut learning: the loss gradients flow backward through induced state inputs, inadvertently leaking future ground truth information directly into the model's own previous predictions. The model exploits these signals to artificially avoid drift, non-causally "regretting" past mistakes rather than learning genuinely reactive recovery. To address this, we introduce a detached receding horizon rollout. By explicitly severing the computation graph between simulation steps, the model learns genuine recovery behaviors from drifted states, forcing it to "rectify" mistakes rather than non-causally optimizing past predictions. Extensive evaluations on the nuScenes and DeepScenario datasets show our approach yields more robust recovery strategies, reducing target collisions by up to 33.24% compared to fully differentiable closed-loop training at high replanning frequencies. Furthermore, compared to standard open-loop baselines, our non-differentiable framework decreases collisions by up to 27.74% in dense environments while simultaneously improving multi-modal prediction diversity and lane alignment.

Figures (3)

• Figure 1: Closed-Loop Sample Generation: Illustration of our sequential simulation rollout. At $t=0$ (Top), the model generates an initial open-loop prediction for the full horizon ($H_{pred}$). Instead of executing this entirely, the simulator advances the target vehicle by a shorter interval, $H_{step}$, to a new physical state (Middle). The model is then queried again from this updated state to produce a new closed-loop prediction for the remaining horizon. This iterative process allows the replanned trajectory to actively realign with the ground truth rather than passively drifting with early forecasting errors (Bottom).
• Figure 2: Closed-Loop State Space Model with Differentiable vs. Non-Differentiable Simulators: A computational flow diagram illustrating the training dynamics. In a fully differentiable setup (red+black dashed lines), gradients backpropagate through the environment simulator, inadvertently allowing future trajectory information to leak into the previous prediction. Our proposed non-differentiable approach explicitly incorporates a gradient detachment step. This severs the gradient flow between the executed trajectory and the subsequent input state (black dashed lines), effectively preventing shortcut learning while preserving the benefits of closed-loop state space exploration.
• Figure 3: Comparison of the open-loop (OL) vs. closed-loop (CL) trained models under various closed-loop evaluation setups with different replanning frequencies, $f_{step} = 6/H_{step}$.