PlanTRansformer: Unified Prediction and Planning with Goal-conditioned Transformer

TL;DR

PlanTRansformer (PTR) addresses the prediction–planning gap in autonomous driving by unifying goal-conditioned prediction with planning constraints in a Transformer framework. It introduces differentiable dynamic feasibility, collision avoidance, reachable-lane routing, and goal-conditioned reasoning via high-level commands, plus a teacher–student strategy to mimic unknown agent intents during inference. On the Waymo Open Motion Dataset, PTR improves marginal and joint prediction mAP by 4.3% and 3.5%, and reduces planning error by 15.5% at 5 seconds compared to GameFormer, while remaining architecture-agnostic for broad applicability. These results demonstrate enhanced safety and feasibility of ego trajectories in interactive scenes, underscoring PTR’s potential to bridge prediction and planning in real-world driving systems.

Abstract

Trajectory prediction and planning are fundamental yet disconnected components in autonomous driving. Prediction models forecast surrounding agent motion under unknown intentions, producing multimodal distributions, while planning assumes known ego objectives and generates deterministic trajectories. This mismatch creates a critical bottleneck: prediction lacks supervision for agent intentions, while planning requires this information. Existing prediction models, despite strong benchmarking performance, often remain disconnected from planning constraints such as collision avoidance and dynamic feasibility. We introduce Plan TRansformer (PTR), a unified Gaussian Mixture Transformer framework integrating goal-conditioned prediction, dynamic feasibility, interaction awareness, and lane-level topology reasoning. A teacher-student training strategy progressively masks surrounding agent commands during training to align with inference conditions where agent intentions are unavailable. PTR achieves 4.3%/3.5% improvement in marginal/joint mAP compared to the baseline Motion Transformer (MTR) and 15.5% planning error reduction at 5s horizon compared to GameFormer. The architecture-agnostic design enables application to diverse Transformer-based prediction models. Project Website: https://github.com/SelzerConst/PlanTRansformer

Figures (5)

• Figure 1: PTR extends prediction models with planning constrains and goal conditioning through command guidance to bridge the prediction-planning asymmetry. Left: Surrounding agents with unknown intentions produce ambiguous multimodal predictions. Right: Ego vehicle with explicit navigation context generates deterministic, goal-aligned trajectories.
• Figure 2: The PTR framework processes three input modalities (agent history, map polylines, reachable lanes) through polyline encoders to obtain features $F_A$, $F_M$, $F_L$. A transformer-based scene context encoder with local attention fuses these via $F_{AML}$ and decomposes into refined representations $F_A'$, $F_M'$, $F_L'$. Command-specific intention points initialize the motion decoder's query features, refined through transformer layers with scene features to produce multimodal predictions via GMM.
• Figure 3: The decoder of the PTR framework integrates command-guided query initialization with reachable lane features. Static and dynamic motion queries are initialized from high-level commands and refined iteratively through self-attention and cross-attention mechanisms with scene context and lane features, producing multimodal trajectory predictions via GMM.
• Figure 4: Qualitative top-3 prediction comparison of MTR (top) and PTR (bottom) across five scenarios. Scenario visualization shows agents of interest (pink boxes), other agents (blue boxes), prediction (blue lines), ground-truth (white dashed lines), lane centerline (gray dotted lines), lane separator (gray dashed lines), and road boarder (black solid lines). a) Vehicle (1) exhibits more confident left-turn predictions centered on lane markings in PTR; cyclist (2) has less GT alignment due to the near-miss interaction. b) PTR increases left-turn confidence for vehicle (1) through high-level command guidance, while vehicle (2) maintains multimodal predictions exploring alternative routes. c) PTR shows improved centerline adherence and GT alignment for vehicle (2); vehicles (1) and (3) generate more conservative trajectories due to collision constraints, deviating from GT endpoints for vehicle (1) while vehicle (3) deviates less. d) Vehicle (1) achieves better centerline alignment in PTR, improving map consistency but reducing GT fidelity in non-compliant driving behavior. e) PTR produces conservative trajectories for vehicle (1), while vehicle (2) continues nominal behavior without significant collision-avoidance influences.
• Figure 5: Qualitative top-1 planning comparison of GameFormer (top) and PTR (bottom) across three scenarios, demonstrating planning capabilities and high-level command influence (last pair). Scenario visualization shows ego (red box), agents of interest (pink boxes), other agents (blue boxes), prediction (blue lines), plan (red line), ground-truth (white dashed lines), lane centerline (gray dotted lines), lane separator (gray dashed lines), and road boarder (black solid lines). a) Ego (1) generates a similar trajectory as GameFormer, while surrounding vehicle (2) maintains a more feasible path. b) In a dense interactive scenario with nearby vehicles (2, 3, 4), ego (1) produces more conservative GT aligned planning trajectory in PTR compared to GameFormer. c) PTR's ego (1) planning aligns with the left-turn high-level command, while GameFormer's trajectory appears misaligned with the scenario intent. d) Same scenario as c) both images generated by PTR with modified commands (top: right turn; bottom: straight), demonstrating PTR's command-conditioned planning responsiveness to navigation guidance.