Self-Imitated Diffusion Policy for Efficient and Robust Visual Navigation

TL;DR

The paper tackles suboptimality and inefficiency in imitation-learning diffusion policies for visual navigation by introducing SIDP, a reward-guided self-imitation framework that concentrates learning on high-quality, self-generated trajectories. By recasting training as distribution matching and employing a reward-weighted denoising loss, SIDP avoids backpropagation through time and eliminates external trajectory selectors, enabling end-to-end and faster inference with DDIM. The approach is augmented with goal-agnostic exploration and a reward-driven curriculum to diversify data and focus learning on informative scenarios. Empirical results show state-of-the-art SR and SPL on high-fidelity benchmarks and a 2.5× speedup on edge devices (Jetson Orin Nano), with successful real-world deployments across multiple robots, underscoring SIDP’s practicality and robustness in real-world navigation tasks.

Abstract

Diffusion policies (DP) have demonstrated significant potential in visual navigation by capturing diverse multi-modal trajectory distributions. However, standard imitation learning (IL), which most DP methods rely on for training, often inherits sub-optimality and redundancy from expert demonstrations, thereby necessitating a computationally intensive "generate-then-filter" pipeline that relies on auxiliary selectors during inference. To address these challenges, we propose Self-Imitated Diffusion Policy (SIDP), a novel framework that learns improved planning by selectively imitating a set of trajectories sampled from itself. Specifically, SIDP introduces a reward-guided self-imitation mechanism that encourages the policy to consistently produce high-quality trajectories efficiently, rather than outputs of inconsistent quality, thereby reducing reliance on extensive sampling and post-filtering. During training, we employ a reward-driven curriculum learning paradigm to mitigate inefficient data utility, and goal-agnostic exploration for trajectory augmentation to improve planning robustness. Extensive evaluations on a comprehensive simulation benchmark show that SIDP significantly outperforms previous methods, with real-world experiments confirming its effectiveness across multiple robotic platforms. On Jetson Orin Nano, SIDP delivers a 2.5$\times$ faster inference than the baseline NavDP, i.e., 110ms VS 273ms, enabling efficient real-time deployment.

Figures (7)

• Figure 1: Comparison between imitation and self-imitation based diffusion policy. (a) The classic imitation-based diffusion policy tends to generate trajectories with high variability, including unsafe, non-shortest or goal-unreachable routes, therefore relies on extensive sampling with post-filtering to obtain a final plan. (b) SIDP achieves distribution concentration via reward-guided self-imitation, enabling robust and efficient path planning without auxiliary selectors. (c) Validation across two robot platforms in real-world environments.
• Figure 2: Overview of the Self-Imitated Diffusion Policy (SIDP). The framework generates candidate trajectories using the current policy $\pi_\theta$ and filters them via a reward-based sorting gate. High-reward samples are then used to compute the weighted denoising loss $\mathcal{L}_{\text{SIDP}}$, updating the policy parameters to iteratively align with the optimal distribution.
• Figure 3: Learning curves of SIDP under different temperature coefficients during training. The curves are smoothed using a Gaussian filter ($\sigma=10$), and the shaded region indicates the rolling-window standard deviation, capturing performance fluctuations caused by environment variations and randomly sampled navigation goals.
• Figure 4: Ablation study of the reward-guided self-imitation mechanism and the Softmax temperature coefficient $\tau$.
• Figure 5: Ablation study of goal-agnostic training under different evaluation settings.