Risk Map As Middleware: Towards Interpretable Cooperative End-to-end Autonomous Driving for Risk-Aware Planning

TL;DR

This work tackles occlusion-driven risk and interpretability in end-to-end autonomous driving by introducing Risk Map as Middleware (RiskMM), a cooperative framework that learns a spatiotemporal risk map from multi-agent representations and uses it to condition a differentiable, learning-based MPC planner with explicit physical constraints. The methodology combines V2X-assisted scenario awareness, occupancy/risk prediction via attention-based fusion, and a quadratic-cost MPC whose weights are derived from the learned risk map, enabling transparent interpretation of planning decisions. Evaluations on the V2XPnP-Seq dataset show that RiskMM improves risk-aware planning, provides interpretable planner behavior through learned MPC weights, and maintains robustness to communication noise; ablations confirm the critical roles of the risk map and modular training. Overall, RiskMM advances interpretable, risk-aware cooperative driving by uniting explicit risk modeling with end-to-end optimization, with potential for broader adoption across vehicle types and scenarios.

Abstract

End-to-end paradigm has emerged as a promising approach to autonomous driving. However, existing single-agent end-to-end pipelines are often constrained by occlusion and limited perception range, resulting in hazardous driving. Furthermore, their black-box nature prevents the interpretability of the driving behavior, leading to an untrustworthiness system. To address these limitations, we introduce Risk Map as Middleware (RiskMM) and propose an interpretable cooperative end-to-end driving framework. The risk map learns directly from the driving data and provides an interpretable spatiotemporal representation of the scenario from the upstream perception and the interactions between the ego vehicle and the surrounding environment for downstream planning. RiskMM first constructs a multi-agent spatiotemporal representation with unified Transformer-based architecture, then derives risk-aware representations by modeling interactions among surrounding environments with attention. These representations are subsequently fed into a learning-based Model Predictive Control (MPC) module. The MPC planner inherently accommodates physical constraints and different vehicle types and can provide interpretation by aligning learned parameters with explicit MPC elements. Evaluations conducted on the real-world V2XPnP-Seq dataset confirm that RiskMM achieves superior and robust performance in risk-aware trajectory planning, significantly enhancing the interpretability of the cooperative end-to-end driving framework. The codebase will be released to facilitate future research in this field.

Figures (5)

• Figure 1: Comparison of different autonomous driving paradigms. (a) Modular paradigm, which suffers from error propagation and the limited perception of single-vehicle systems. (b) Conventional end-to-end paradigm, which directly maps the single-vehicle sensor input to control commands but lacks interpretability and holistic perception. (c) Proposed end-to-end paradigm, which introduces the risk map as the middleware and integrates the multi-agent cooperation to enhance interpretability and overcome perception constraints.
• Figure 2: Overview of the proposed RiskMM pipeline. (i) Scenario Awareness: capturing a multi-agent spatiotemporal representation via auxiliary cooperative perception and prediction tasks; (ii) Risk Recognition and Trajectory Planning: explicitly modeling the driving risk as a risk map and subsequently generating the planning trajectory with the learning-based MPC module.
• Figure 3: Qualitative results of RiskMM in the real-world V2XPnP-Seq Dataset. The first and second rows show the performance in the first scenario, while the third and fourth rows illustrate the performance in the second scenario. The risk map aligns the information of the accurate detection and prediction, and introduces the interaction information to guide the planning.
• Figure 4: Learned weights of the proposed MPC planner. For Scenario 1, the quadratic term weight related to speed is relatively lower, while the linear term weight related to speed is relatively higher compared to Scenario 2.
• Figure 5: Communication noise and delay experiment.