SPACeR: Self-Play Anchoring with Centralized Reference Models

TL;DR

SPACeR tackles the challenge of producing human-like, reactive autonomous-driving policies that scale to multi-agent self-play. It introduces a centralized pretrained reference tokenized model that provides per-timestep likelihoods and KL-based alignment to steer decentralized self-play without relying on ground-truth trajectories. The approach yields competitive realism compared to imitation-learning methods while delivering substantial gains in inference speed and parameter efficiency. This enables fast, real-time multi-agent simulation and scalable closed-loop planner evaluation, advancing AV testing and development.

Abstract

Developing autonomous vehicles (AVs) requires not only safety and efficiency, but also realistic, human-like behaviors that are socially aware and predictable. Achieving this requires sim agent policies that are human-like, fast, and scalable in multi-agent settings. Recent progress in imitation learning with large diffusion-based or tokenized models has shown that behaviors can be captured directly from human driving data, producing realistic policies. However, these models are computationally expensive, slow during inference, and struggle to adapt in reactive, closed-loop scenarios. In contrast, self-play reinforcement learning (RL) scales efficiently and naturally captures multi-agent interactions, but it often relies on heuristics and reward shaping, and the resulting policies can diverge from human norms. We propose SPACeR, a framework that leverages a pretrained tokenized autoregressive motion model as a centralized reference policy to guide decentralized self-play. The reference model provides likelihood rewards and KL divergence, anchoring policies to the human driving distribution while preserving RL scalability. Evaluated on the Waymo Sim Agents Challenge, our method achieves competitive performance with imitation-learned policies while being up to 10x faster at inference and 50x smaller in parameter size than large generative models. In addition, we demonstrate in closed-loop ego planning evaluation tasks that our sim agents can effectively measure planner quality with fast and scalable traffic simulation, establishing a new paradigm for testing autonomous driving policies.

Figures (4)

• Figure 1: Overview of SPACeR Framework. Self-play reinforcement learning is anchored to a pretrained tokenized reference model $\pi_{\text{ref}}$, which provides a human-likeness distributional signal. The self-play policy $\pi_{\theta}$ is decentralized and conditioned on local observations, whereas $\pi_{\text{ref}}$ is centralized and conditioned on the full scene context.
• Figure 2: Correlation Coefficients of PDM Scores Across Policy Evaluation Strategies. Using our approach leads to consistently lower correlations with ground-truth log replays across all metrics, suggesting more realistic penalization of unsafe policies—especially for collisions.
• Figure 3: Example scenarios where WOSAC metrics produce unrealistic estimates
• Figure 4: Additional qualitative examples of WOSAC limitations. Top row: due to sensor noise, the ground-truth agents collided in the data; consequently, safe simulated behaviors are assigned low collision likelihood. Bottom row: the ground-truth agent collided with the road edge, so remaining on-road is given near-zero off-road likelihood and a low map score. These cases illustrate how WOSAC can penalize safe and realistic behaviors when they diverge from noisy or imperfect logged trajectories.