Getting SMARTER for Motion Planning in Autonomous Driving Systems

TL;DR

This work addresses the challenge of robustly evaluating motion planning in autonomous driving by leveraging a scalable, realistic simulator. It introduces SMARTS 2.0, which integrates real-world data replay, sensor modeling, V2V communication, diagnostics, and real-time execution to enable large-scale, heterogeneous agent simulation. The authors propose a novel Interactive Motion Planning Benchmark with collaborative and adaptive planning scenarios, accompanied by a unified metric framework that includes both common and task-specific measures, such as $S_{final}$ and $S_{bench}$. Empirical results show substantial performance gains of SMARTS 2.0 over the previous version and reveal varying strengths across SOTA planners, underscoring the complexity of balancing safety, efficiency, humanness, and rule compliance in real-world driving. The framework lays groundwork for future research in connected driving, richer traffic scenarios, and more restrictive evaluation criteria that better reflect real-world constraints.

Abstract

Motion planning is a fundamental problem in autonomous driving and perhaps the most challenging to comprehensively evaluate because of the associated risks and expenses of real-world deployment. Therefore, simulations play an important role in efficient development of planning algorithms. To be effective, simulations must be accurate and realistic, both in terms of dynamics and behavior modeling, and also highly customizable in order to accommodate a broad spectrum of research frameworks. In this paper, we introduce SMARTS 2.0, the second generation of our motion planning simulator which, in addition to being highly optimized for large-scale simulation, provides many new features, such as realistic map integration, vehicle-to-vehicle (V2V) communication, traffic and pedestrian simulation, and a broad variety of sensor models. Moreover, we present a novel benchmark suite for evaluating planning algorithms in various highly challenging scenarios, including interactive driving, such as turning at intersections, and adaptive driving, in which the task is to closely follow a lead vehicle without any explicit knowledge of its intention. Each scenario is characterized by a variety of traffic patterns and road structures. We further propose a series of common and task-specific metrics to effectively evaluate the performance of the planning algorithms. At the end, we evaluate common motion planning algorithms using the proposed benchmark and highlight the challenges the proposed scenarios impose. The new SMARTS 2.0 features and the benchmark are publicly available at github.com/huawei-noah/SMARTS.

Figures (8)

• Figure 1: Left to right: bird-eye-view image of the simulation environment, and the visualization of the 2D sensor observation space. The mission vehicle is centered in the map and the blue circle shows the maximum observation range of the sensor.
• Figure 2: Overview of route selection GUI. The blue lane segment is selected by user for additional analysis. Text is enhanced from the original for better visibility.
• Figure 3: Various types of turn maneuvers for collaborative planning.
• Figure 4: Vehicle-following scenarios for adaptive planning.
• Figure 5: Driving comfort $comf$ based on travel time to the destination ($T_{trv}$), uncomfortable period ($T_u$), and penalty period ($T_p$).