RoDiF: Robust Direct Fine-Tuning of Diffusion Policies with Corrupted Human Feedback
Amitesh Vatsa, Zhixian Xie, Wanxin Jin
TL;DR
RoDiF tackles the challenge of refining diffusion policies with human preferences when feedback is noisy or corrupted. By formulating a Unified MDP that couples the diffusion denoising chain with environmental dynamics and reinterpreting DPO through a conservative, hypothesis-cutting lens, RoDiF achieves robust reward-free fine-tuning without assuming a noise distribution. The approach demonstrates strong alignment and high task success across multiple diffusion backbones on long-horizon manipulation tasks, maintaining performance under up to 30% corrupted labels. This work advances practical, safe, and reliable reinforcement-like learning for diffusion-based robotic control by mitigating brittle updates caused by inconsistent human feedback.
Abstract
Diffusion policies are a powerful paradigm for robotic control, but fine-tuning them with human preferences is fundamentally challenged by the multi-step structure of the denoising process. To overcome this, we introduce a Unified Markov Decision Process (MDP) formulation that coherently integrates the diffusion denoising chain with environmental dynamics, enabling reward-free Direct Preference Optimization (DPO) for diffusion policies. Building on this formulation, we propose RoDiF (Robust Direct Fine-Tuning), a method that explicitly addresses corrupted human preferences. RoDiF reinterprets the DPO objective through a geometric hypothesis-cutting perspective and employs a conservative cutting strategy to achieve robustness without assuming any specific noise distribution. Extensive experiments on long-horizon manipulation tasks show that RoDiF consistently outperforms state-of-the-art baselines, effectively steering pretrained diffusion policies of diverse architectures to human-preferred modes, while maintaining strong performance even under 30% corrupted preference labels.