Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Differentiable Integrated Motion Prediction and Planning with Learnable Cost Function for Autonomous Driving

Zhiyu Huang, Haochen Liu, Jingda Wu, Chen Lv

TL;DR

<3-5 sentence high-level summary> The paper tackles the problem of decoupled prediction and planning in autonomous driving, where tuning planning costs is difficult and safety can be compromised. It introduces Differentiable Integrated Prediction and Planning (DIPP), a framework that jointly learns prediction, a learnable cost function, and a differentiable motion planner, enabling end-to-end optimization driven by planning performance. The approach uses a Transformer-based multi-agent predictor to generate planning-aware, multi-modal futures and a differentiable Gauss-Newton planner with a kinematic bicycle model, all trained with four losses on a large real-world dataset. Experiments on the Waymo Open Motion Dataset show improvements in safety, comfort, and human-likeness in both open-loop and closed-loop settings, with ablations confirming the importance of each learnable component and the benefits of joint training over separated pipelines.

Abstract

Predicting the future states of surrounding traffic participants and planning a safe, smooth, and socially compliant trajectory accordingly is crucial for autonomous vehicles. There are two major issues with the current autonomous driving system: the prediction module is often separated from the planning module and the cost function for planning is hard to specify and tune. To tackle these issues, we propose a differentiable integrated prediction-planning framework (DIPP) that can also learn the cost function from data. Specifically, our framework uses a differentiable nonlinear optimizer as the motion planner, which takes as input the predicted trajectories of surrounding agents given by the neural network and optimizes the trajectory for the autonomous vehicle, enabling all operations to be differentiable, including the cost function weights. The proposed framework is trained on a large-scale real-world driving dataset to imitate human driving trajectories in the entire driving scene and validated in both open-loop and closed-loop manners. The open-loop testing results reveal that the proposed method outperforms the baseline methods across a variety of metrics and delivers planning-centric prediction results, allowing the planning module to output trajectories close to those of human drivers. In closed-loop testing, the proposed method outperforms various baseline methods, showing the ability to handle complex urban driving scenarios and robustness against the distributional shift. Importantly, we find that joint training of planning and prediction modules achieves better performance than planning with a separate trained prediction module in both open-loop and closed-loop tests. Moreover, the ablation study indicates that the learnable components in the framework are essential to ensure planning stability and performance.

Differentiable Integrated Motion Prediction and Planning with Learnable Cost Function for Autonomous Driving

TL;DR

<3-5 sentence high-level summary> The paper tackles the problem of decoupled prediction and planning in autonomous driving, where tuning planning costs is difficult and safety can be compromised. It introduces Differentiable Integrated Prediction and Planning (DIPP), a framework that jointly learns prediction, a learnable cost function, and a differentiable motion planner, enabling end-to-end optimization driven by planning performance. The approach uses a Transformer-based multi-agent predictor to generate planning-aware, multi-modal futures and a differentiable Gauss-Newton planner with a kinematic bicycle model, all trained with four losses on a large real-world dataset. Experiments on the Waymo Open Motion Dataset show improvements in safety, comfort, and human-likeness in both open-loop and closed-loop settings, with ablations confirming the importance of each learnable component and the benefits of joint training over separated pipelines.

Abstract

Predicting the future states of surrounding traffic participants and planning a safe, smooth, and socially compliant trajectory accordingly is crucial for autonomous vehicles. There are two major issues with the current autonomous driving system: the prediction module is often separated from the planning module and the cost function for planning is hard to specify and tune. To tackle these issues, we propose a differentiable integrated prediction-planning framework (DIPP) that can also learn the cost function from data. Specifically, our framework uses a differentiable nonlinear optimizer as the motion planner, which takes as input the predicted trajectories of surrounding agents given by the neural network and optimizes the trajectory for the autonomous vehicle, enabling all operations to be differentiable, including the cost function weights. The proposed framework is trained on a large-scale real-world driving dataset to imitate human driving trajectories in the entire driving scene and validated in both open-loop and closed-loop manners. The open-loop testing results reveal that the proposed method outperforms the baseline methods across a variety of metrics and delivers planning-centric prediction results, allowing the planning module to output trajectories close to those of human drivers. In closed-loop testing, the proposed method outperforms various baseline methods, showing the ability to handle complex urban driving scenarios and robustness against the distributional shift. Importantly, we find that joint training of planning and prediction modules achieves better performance than planning with a separate trained prediction module in both open-loop and closed-loop tests. Moreover, the ablation study indicates that the learnable components in the framework are essential to ensure planning stability and performance.
Paper Structure (29 sections, 14 equations, 7 figures, 8 tables, 1 algorithm)

This paper contains 29 sections, 14 equations, 7 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-agent prediction
  4. Motion planning
  5. Joint prediction and planning
  6. Joint prediction and planning
  7. Method
  8. Problem formulation
  9. Multi-agent prediction and initial plan
  10. Differentiable motion planning
  11. Problem statement
  12. Kinematic model
  13. Cost function
  14. Differentiable optimization
  15. Learning process
  16. ...and 14 more sections

Figures (7)

  • Figure 1: Three different motion planning paradigms: (a) traditional sequential prediction and planning; (b) end-to-end method; (c) our proposed method.
  • Figure 2: The proposed differentiable integrated prediction and planning framework. The neural network predictor is utilized to predict the future states of surrounding agents and the initial plan for the motion planner, and the differentiable motion planner with a learnable cost function is employed to explicitly plan an AV trajectory. All the components are connected and end-to-end differentiable.
  • Figure 3: Illustration of the calculation of safe distance. Other agents are first projected to the Frenet frame to find the interacting agents, and the calculation of distances is in the Cartesian frame.
  • Figure 4: The multi-modal predictions given by the neural network predictor. The trajectories of the ten nearest agents to the AV are predicted by the neural network.
  • Figure 5: Qualitative results of the proposed framework in open-loop testing. The colored solid lines are the planned or predicted trajectories for AV or surrounding agents, and black dotted lines are the ground truth trajectories.
  • ...and 2 more figures