Model Predictive Simulation Using Structured Graphical Models and Transformers

TL;DR

The paper tackles realistic multi-agent trajectory simulation for road users by hybridizing transformer-based forecasts with a probabilistic graphical model and model predictive control. It introduces Model Predictive Simulation (MPS), which post-processes transformer proposals with priors encoded as a factor graph and refines them via Gauss-Newton-based inference in a two-loop MPC framework. Empirical results on the Waymo Open Sim Agents Challenge show safety improvements and competitive realism, with MPS ranking near the top and outperforming the baseline in collision-related metrics. The approach is model-agnostic with respect to the underlying forecast model and requires no additional training, underscoring the value of combining principled priors with data-driven predictions for robust autonomous driving simulation and planning.

Abstract

We propose an approach to simulating trajectories of multiple interacting agents (road users) based on transformers and probabilistic graphical models (PGMs), and apply it to the Waymo SimAgents challenge. The transformer baseline is based on the MTR model, which predicts multiple future trajectories conditioned on the past trajectories and static road layout features. We then improve upon these generated trajectories using a PGM, which contains factors which encode prior knowledge, such as a preference for smooth trajectories, and avoidance of collisions with static obstacles and other moving agents. We perform (approximate) MAP inference in this PGM using the Gauss-Newton method. Finally we sample $K=32$ trajectories for each of the $N \sim 100$ agents for the next $T=8 Δ$ time steps, where $Δ=10$ is the sampling rate per second. Following the Model Predictive Control (MPC) paradigm, we only return the first element of our forecasted trajectories at each step, and then we replan, so that the simulation can constantly adapt to its changing environment. We therefore call our approach "Model Predictive Simulation" or MPS. We show that MPS improves upon the MTR baseline, especially in safety critical metrics such as collision rate. Furthermore, our approach is compatible with any underlying forecasting model, and does not require extra training, so we believe it is a valuable contribution to the community.

Figures (11)

• Figure 1: Factor Graph for $N=2$ agents unrolled for $T$ planning steps. Circles are random variables, gray squares are fixed factors.
• Figure 2: Joint probability model.
• Figure 3: MPS creates diverse (multi-modal) rollouts.
• Figure 4: MPS creates realistic traffic patterns.
• Figure 5: Three simulated scenarios (top to bottom) at different timesteps (left to right) showcasing multi-modal behavior of agents. In the top and bottom simulation, the dark green car takes the left turn. However, in the middle simulation, it turns right. The green car, in the top and middle simulation, attempts the lane change to the left as the cars in front wait at the signal. In the middle simulation, the same green car comes to a stop in the same lane behind the traffic.