PlannerRFT: Reinforcing Diffusion Planners through Closed-Loop and Sample-Efficient Fine-Tuning

TL;DR

PlannerRFT addresses the challenge of robust, multi-modal, scenario-adaptive diffusion-based motion planning by introducing a closed-loop reinforcement fine-tuning framework. It combines policy-guided denoising to sustain diversity with a dual-branch optimization (PPO for exploration and GRPO for trajectory refinement) and leverages nuMax, a GPU-accelerated simulator, for scalable rollouts. The approach yields state-of-the-art results on nuPlan benchmarks, with safer, more efficient, and human-like driving behaviors emerging during learning. This framework advances practical RL-tuning of diffusion planners, enabling more reliable deployment in dynamic road environments while highlighting avenues for extending to sensory-rich or end-to-end planning systems.

Abstract

Diffusion-based planners have emerged as a promising approach for human-like trajectory generation in autonomous driving. Recent works incorporate reinforcement fine-tuning to enhance the robustness of diffusion planners through reward-oriented optimization in a generation-evaluation loop. However, they struggle to generate multi-modal, scenario-adaptive trajectories, hindering the exploitation efficiency of informative rewards during fine-tuning. To resolve this, we propose PlannerRFT, a sample-efficient reinforcement fine-tuning framework for diffusion-based planners. PlannerRFT adopts a dual-branch optimization that simultaneously refines the trajectory distribution and adaptively guides the denoising process toward more promising exploration, without altering the original inference pipeline. To support parallel learning at scale, we develop nuMax, an optimized simulator that achieves 10 times faster rollout compared to native nuPlan. Extensive experiments shows that PlannerRFT yields state-of-the-art performance with distinct behaviors emerging during the learning process.

Figures (16)

• Figure 1: Comparison to Denoising Strategies across various diffusion planning paradigms. (a) Vanilla diffusion planners suffer from mode collapse, offering limited exploration. (b) Anchor-based methods are oriented towards scenario-agnostic actions, leading to noisy interactions. (c) Our policy-guided denoising enables both multi-modal and scenario-adaptive sampling, yielding stable and efficient exploration for optimization.
• Figure 2: Overview of PlannerRFT. We enhance multi-modality during RL sampling through Guided Denoising, with guidance scales modulated by the Exploration Policy to generate scenario-adaptive trajectories (\ref{['sec:exploration']}). The planner gathers on-policy interaction data through Closed-Loop Rollout in simulation (\ref{['sec:rollout']}). A dual-branch optimization framework performs Trajectory Optimization and Exploration Optimization to steer the denoising process (\ref{['sec:optim']}).
• Figure 3: Illustration of nuMax.(a) Scenario cache: nuPlan scenarios are preprocessed and cached for fast loading during large-scale RL rollouts; (b) LQR tracker and scorer: vehicle kinematics and reward computation are calibrated to match nuPlan; and (c) Distributed RL training pipeline: enables communication between PyTorch DistributedDataParallel (DDP) workers and the JAX-based simulator.
• Figure 4: Qualitative Comparison of Pretrained Planner and RFT Planner. In each frame shot, the simulation position and planning trajectory are marked as orange, the ground-truth position and ground-truth trajectory recorded in the driving log are marked as gray and blue, respectively.
• Figure 5: Visualization of Different Exploration Policies.(a) Without guidance: denoising from random noise. (b) Uniform exploration policy: $(\eta_{\text{lat.}}, \eta_{\text{lon.}})$ are sampled from a uniform distribution. (c) Fixed exploration policy: $(\eta_{\text{lat.}}, \eta_{\text{lon.}})$ are sampled from the non-learnable Beta distribution initialized from the Exploration Policy’s zero parameters. (d) Our policy-guided denoising: exploration adapt to the driving context.