When Planners Meet Reality: How Learned, Reactive Traffic Agents Shift nuPlan Benchmarks

TL;DR

This work demonstrates that traditional IDM-based closed-loop evaluations in nuPlan can overstate planner performance due to passive traffic agents. By integrating SMART, a learned reactive traffic model, the authors reveal richer interactions, more challenging scenarios, and a shifted understanding of planner capabilities, introducing the SMART-based CLS-SR benchmark to replace or augment IDM in evaluations. Evaluating 14 planners across Val14, Test14-hard, and interPlan shows that imitation-learned methods degrade in simple settings while rule-based and hybrid methods resist hard interactions; learned planners trained in closed-loop are more stable but can abruptl y fail beyond their training distribution, with CaRL performing best under SMART. The study releases a drop-in SMART integration for nuPlan and highlights the practical impact of realism in traffic agents for benchmarking, suggesting future work on training with SMART and exploring non-deterministic agent behavior.

Abstract

Planner evaluation in closed-loop simulation often uses rule-based traffic agents, whose simplistic and passive behavior can hide planner deficiencies and bias rankings. Widely used IDM agents simply follow a lead vehicle and cannot react to vehicles in adjacent lanes, hindering tests of complex interaction capabilities. We address this issue by integrating the state-of-the-art learned traffic agent model SMART into nuPlan. Thus, we are the first to evaluate planners under more realistic conditions and quantify how conclusions shift when narrowing the sim-to-real gap. Our analysis covers 14 recent planners and established baselines and shows that IDM-based simulation overestimates planning performance: nearly all scores deteriorate. In contrast, many planners interact better than previously assumed and even improve in multi-lane, interaction-heavy scenarios like lane changes or turns. Methods trained in closed-loop demonstrate the best and most stable driving performance. However, when reaching their limits in augmented edge-case scenarios, all learned planners degrade abruptly, whereas rule-based planners maintain reasonable basic behavior. Based on our results, we suggest SMART-reactive simulation as a new standard closed-loop benchmark in nuPlan and release the SMART agents as a drop-in alternative to IDM at https://github.com/shgd95/InteractiveClosedLoop.

Figures (4)

• Figure 1: Intelligent Driver Model (IDM) vs. SMART agents: same scene, different world. Starting with the same real-world scene state, the two traffic agent models lead to different scene developments when evaluating the same ego trajectory planner in a closed-loop simulation. Since the IDM cannot see vehicles in adjacent lanes it does not react to the ego vehicle's lane change attempt and behaves passively. In contrast, SMART agents trained from real traffic data behave less passive, react to other agents across lanes and execute driving maneuvers themselves. These characteristics of the sim agents are important for testing complex interactions and for assessing a planner’s driving behavior in real-world traffic.
• Figure 2: Automated Driving Evaluation. In open-loop evaluation the planner generates a future trajectory from a real observation, which is then compared to an expert's driving log. However, open-loop performance does not correlate with real-world driving performance as it lacks evidence for the system's behavior when deviating from human expert states. Closed-loop evaluation addresses this shortcoming by allowing the scene state to deviate from the recorded real-world data and rewards good driving behavior instead of imitation. Nevertheless, the quality of the traffic agent model determines how well closed-loop performance mirrors real-world driving.
• Figure 3: Closed-loop simulation with SMART agents compared to IDM agents in two exemplary scenarios of interPlan-lane-change. In scenario I the ego (red) must execute multiple lane changes in dense traffic to follow the intended route (dashed). SMART agents show more diverse behavior and decreased passivity compared to IDM. The ego surprises the other vehicle with its sudden lane change inside the intersection, causing a collision. In scenario II a SMART agent turns right and then stops at the side of the road. The ego vehicle nudges around it after turning, proving its interaction capability, which remains untested when using IDM agents. In all four simulations the ego vehicle is steered by the best performing planner CaRL.
• Figure 4: Planner performance shift under SMART agents compared to IDM agents on Val14 (CLS-SR $-$ CLS-R). The left plot shows a metric-wise breakdown per planner. The right plot provides a scenario-wise analysis.