Variance-Reduced Model Predictive Path Integral via Quadratic Model Approximation

TL;DR

This work tackles high variance and limited sample efficiency in sampling-based MPPI by introducing a model-guided, variance-reduced framework. It decomposes the objective into a known model $m^k$ and a residual $r^k$, achieving a closed-form Gaussian prior when the model is quadratic, and samples the residual around this prior to update the mean. The approach supports exact, Gauss-Newton, quasi-Newton, and randomized smoothing for model construction, and includes stability mechanisms such as Polyak averaging and a variance lower bound. Empirical results across static benchmarks, a nonlinear cart-pole task, and a contact-rich manipulation problem show faster convergence, higher ESS, and robustness in low-sample regimes compared to vanilla MPPI and CMA-ES, suggesting practical improvements for sample-efficient, real-time control.

Abstract

Sampling-based controllers, such as Model Predictive Path Integral (MPPI) methods, offer substantial flexibility but often suffer from high variance and low sample efficiency. To address these challenges, we introduce a hybrid variance-reduced MPPI framework that integrates a prior model into the sampling process. Our key insight is to decompose the objective function into a known approximate model and a residual term. Since the residual captures only the discrepancy between the model and the objective, it typically exhibits a smaller magnitude and lower variance than the original objective. Although this principle applies to general modeling choices, we demonstrate that adopting a quadratic approximation enables the derivation of a closed-form, model-guided prior that effectively concentrates samples in informative regions. Crucially, the framework is agnostic to the source of geometric information, allowing the quadratic model to be constructed from exact derivatives, structural approximations (e.g., Gauss- or Quasi-Newton), or gradient-free randomized smoothing. We validate the approach on standard optimization benchmarks, a nonlinear, underactuated cart-pole control task, and a contact-rich manipulation problem with non-smooth dynamics. Across these domains, we achieve faster convergence and superior performance in low-sample regimes compared to standard MPPI. These results suggest that the method can make sample-based control strategies more practical in scenarios where obtaining samples is expensive or limited.

Figures (6)

• Figure 1: Illustration of covariance adaptation via Newton-like approximation. Model-guided MPPI and vanilla MPPI start at the same state (circle) and use $100$ samples from an isotropic Gaussian prior. The cross indicates the vanilla MPPI update. Our method exploits the gradient and Hessian at $\bar{x}^0$ to construct a quadratic model. This leads to a guided prior centered at $\tilde{x}^0$ (rectangle), concentrating the sampling distribution (yellow) along the valley and enabling an update $\bar{x}^1$ (triangle) by sampling the residual toward the optimum.
• Figure 2: Comparison coarse vs fine model. (a) A large smoothing kernel $\sigma = 1.0$ suppresses sinusoidal disturbances (grey), yielding a coarse approximation (blue) capturing global geometry. The resulting quadratic model (red) generates a proposal $\tilde{x}^1$ moving toward the global minimum despite local nonconvexities. (b) Near the optimum, reducing the scale to $\sigma = 0.1$ captures the local curvature for final convergence.
• Figure 3: Cart-pole swing-up results. Comparison of convergence behavior and variance across sample budgets $N \in \{2, \dots, 1024\}$ and $20$ random seeds.
• Figure 4: Convergence comparison of Hessian approximations on the cart-pole swing-up task. We compare dense curvature models (Analytical, Gauss-Newton, BFGS), against a diagonal approximation (Adam) and the isotropic baseline (Vanilla MPPI). The median distance to the reference optimum is plotted across 10 random seeds, with shaded areas indicating the interquartile range (IQR).
• Figure 5: Performance profile on Single-Finger Sphere Manipulation. Comparison across $200$ randomized tasks.