The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Review

TL;DR

The paper surveys how deep learning-enabled automated driving systems can integrate prediction and planning, arguing that traditional modular pipelines are insufficient for modeling bidirectional interactions with surrounding traffic. It classifies integration strategies into sequential, undirected, and bidirectional IPPS, detailing their architectures, information flows, and behavioral implications, while highlighting data representations, scene coordinates, and output modalities. The authors discuss planning paradigms (cost-based, regression, hybrid), and present extensive discussion on benchmarking, evaluation in open- and closed-loop setups, and the need for interactive scenario benchmarks and V2X-enabled coordination. Key contributions include a taxonomy of IPPS designs, critical analysis of their safety and interaction properties, and a roadmap identifying gaps in benchmarking, simulation realism, and technology integration. The work emphasizes that bidirectional, interaction-aware IPPSs—especially with scenario-based planning and end-to-end differentiable designs—are crucial for safer, more efficient automated driving and outlines concrete directions for future research.

Abstract

Automated driving has the potential to revolutionize personal, public, and freight mobility. Beside accurately perceiving the environment, automated vehicles must plan a safe, comfortable, and efficient motion trajectory. To promote safety and progress, many works rely on modules that predict the future motion of surrounding traffic. Modular automated driving systems commonly handle prediction and planning as sequential, separate tasks. While this accounts for the influence of surrounding traffic on the ego vehicle, it fails to anticipate the reactions of traffic participants to the ego vehicle's behavior. Recent methods increasingly integrate prediction and planning in a joint or interdependent step to model bidirectional interactions. To date, a comprehensive overview of different integration principles is lacking. We systematically review state-of-the-art deep learning-based planning systems, and focus on how they integrate prediction. Different facets of the integration ranging from system architecture to high-level behavioral aspects are considered and related to each other. Moreover, we discuss the implications, strengths, and limitations of different integration principles. By pointing out research gaps, describing relevant future challenges, and highlighting trends in the research field, we identify promising directions for future research.

Figures (7)

• Figure 1: How should the ego vehicle behave? Various behaviors are possible for the ego vehicle. Here, we depict two in red and orange. Their consequences depend on the behavior of the observed surrounding vehicle, which is depicted by blue arrows (prediction A or prediction B). At the same time, the surrounding vehicle might react to the ego vehicle's action, i.e. the surrounding vehicle's behavior also depends on the ego vehicle's decision. Different methods exist to forecast the behavior of the surrounding vehicle, and various ways exist to leverage this to decide on a safe and goal-directed plan for the ego-vehicle. In this survey, we systematically categorize and review methods that integrate prediction and planning for self-driving vehicles. We highlight their capabilities and limitations and discuss prospects for future research.
• Figure 2: Overview of automated driving systems. There are three system architectures shown on the left. Modular systems consist of individual modules, whose interfaces provide interpretability but restrict information flow and end-to-end differentiability. In contrast, monolithic E2E systems are end-to-end differentiable but not interpretable. Modular E2E systems combine both properties: they are end-to-end differentiable and interpretable. All three system architectures comprise an integrated prediction and planning system (IPPS), depicted in green. Our work focuses on this very part and identifies three paradigms to integrate prediction and planning, as shown on the right. Sequential IPPSs condition one task on the other. Undirected IPPSs allow for more complex interactions but provide low interpretability. Bidirectional IPPSs explicitly ensure that both tasks are mutually conditioned on each other. Our review analyzes these integration paradigms and highlights their compatibility with different system architectures.
• Figure 3: An exemplary sequential IPPS that falls in the human leader category. First, the SV's future is predicted, and then a plan is made based on the anticipated future. In this example, the planning step involves evaluating three candidate plans and selecting the one that does not collide with the predicted SV positions.
• Figure 4: An exemplary robot leader IPPS. First, three candidate plans are generated, e.g., with a kinematic sampler. Then, a prediction conditioned on each candidate plan is inferred. In the example, the EV expects the SV to brake for it if it slowly merges to the left lane. Thus, it opts for this plan. Selecting this plan without hedging against the risk that the EV does not brake as expected results in overconfident behavior.
• Figure 5: An exemplary joint optimization IPPS. A joint global cost function is optimized by selecting the best of several potential scenarios. Therefore, a joint prediction and planning step forecasts several potential scenarios that describe the EV and SV behavior. Then, each of them is evaluated using a global cost function that evaluates the outcome for all vehicles. Here, a plan for the EV is selected that corresponds to the scenario where the SV brakes to let the EV merge before it.