Unifying Model Predictive Path Integral Control, Reinforcement Learning, and Diffusion Models for Optimal Control and Planning
Yankai Li, Mo Chen
TL;DR
The paper unifies Model Predictive Path Integral (MPPI) control, Reinforcement Learning, and diffusion-based planning under gradient ascent on the Gibbs measure by introducing a Gaussian-smoothed energy framework. It shows MPPI is equivalent to optimizing a smoothed energy $\widetilde{E}(U)$, that Policy Gradient with an exponential objective aligns with MPPI under a fixed initial state, and that diffusion models share the same update structure through their gradient terms, aligning reverse diffusion with MPPI updates. A key contribution is removing the need for cost decomposition and revealing exact connections among these methods via a common energy-based view. The work also notes how guided diffusion-based planning blends data priors with objective-driven updates, bridging learning from demonstrations with optimization-based control. This unified perspective enhances understanding of planning under uncertainty and suggests cross-method transfer of ideas for efficient trajectory optimization.
Abstract
Model Predictive Path Integral (MPPI) control, Reinforcement Learning (RL), and Diffusion Models have each demonstrated strong performance in trajectory optimization, decision-making, and motion planning. However, these approaches have traditionally been treated as distinct methodologies with separate optimization frameworks. In this work, we establish a unified perspective that connects MPPI, RL, and Diffusion Models through gradient-based optimization on the Gibbs measure. We first show that MPPI can be interpreted as performing gradient ascent on a smoothed energy function. We then demonstrate that Policy Gradient methods reduce to MPPI by applying an exponential transformation to the objective function. Additionally, we establish that the reverse sampling process in diffusion models follows the same update rule as MPPI.