Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Introduction to Zero-Order Optimization Techniques for Robotics

Armand Jordana, Jianghan Zhang, Joseph Amigo, Ludovic Righetti

TL;DR

This work addresses the challenge of derivative-free optimization in robotics by presenting a unified mathematical framework for zero-order methods that connect trajectory optimization and reinforcement learning. It develops a random search-based approach, incorporating Gaussian smoothing and the log-sum-exp MPPI transform, and shows how these ideas subsume and relate to CMA-ES, SPSA, and other gradient-free techniques. The authors demonstrate connections between TO and RL within this framework and derive competitive new RL-style algorithms, highlighting the role of stochasticity in escaping local minima. The results suggest practical benefits for non-differentiable, contact-rich robotics tasks and point toward future directions such as constrained zero-order optimization and scalable parallelization for online and offline settings.

Abstract

Zero-order optimization techniques are becoming increasingly popular in robotics due to their ability to handle non-differentiable functions and escape local minima. These advantages make them particularly useful for trajectory optimization and policy optimization. In this work, we propose a mathematical tutorial on random search. It offers a simple and unifying perspective for understanding a wide range of algorithms commonly used in robotics. Leveraging this viewpoint, we classify many trajectory optimization methods under a common framework and derive novel competitive RL algorithms.

An Introduction to Zero-Order Optimization Techniques for Robotics

TL;DR

This work addresses the challenge of derivative-free optimization in robotics by presenting a unified mathematical framework for zero-order methods that connect trajectory optimization and reinforcement learning. It develops a random search-based approach, incorporating Gaussian smoothing and the log-sum-exp MPPI transform, and shows how these ideas subsume and relate to CMA-ES, SPSA, and other gradient-free techniques. The authors demonstrate connections between TO and RL within this framework and derive competitive new RL-style algorithms, highlighting the role of stochasticity in escaping local minima. The results suggest practical benefits for non-differentiable, contact-rich robotics tasks and point toward future directions such as constrained zero-order optimization and scalable parallelization for online and offline settings.

Abstract

Zero-order optimization techniques are becoming increasingly popular in robotics due to their ability to handle non-differentiable functions and escape local minima. These advantages make them particularly useful for trajectory optimization and policy optimization. In this work, we propose a mathematical tutorial on random search. It offers a simple and unifying perspective for understanding a wide range of algorithms commonly used in robotics. Leveraging this viewpoint, we classify many trajectory optimization methods under a common framework and derive novel competitive RL algorithms.

Paper Structure

This paper contains 29 sections, 56 equations, 7 figures, 1 table, 10 algorithms.

Table of Contents

  1. Introduction
  2. Random search
  3. Simple random search
  4. Pure (or Global) Random Search
  5. Greedy Local Search
  6. Random search via gradient approximation
  7. Random Coordinate Descent
  8. Simultaneous Perturbation Stochastic Approximation (SPSA)
  9. Gaussian Smoothing
  10. Simulated Annealing
  11. Trajectory Optimization
  12. Predictive sampling
  13. The log-sum-exp transform: MPPI
  14. Covariance Matrix Adaptation (CMA)
  15. Numerical experiments
  16. ...and 14 more sections

Figures (7)

  • Figure 1: Randomized smoothing (RS) compared to log-sum-exp (LSE) smoothing (with $\Sigma=I$).
  • Figure 2: Cost according to the number of iterations for different TO test problems. The solid lines represent the median taken over six seeds.
  • Figure 3: Performance of each RL algorithm.
  • Figure 4: Risk-seeking and Risk-averse transformation
  • Figure 5: Cost according to the number of iterations for different test problems. The solid line represents the median taken over 6 seeds.
  • ...and 2 more figures

Theorems & Definitions (9)

  • Remark 1
  • Remark 2
  • Remark 3
  • Remark 4
  • Remark 5
  • Remark 6
  • Remark 7
  • Remark 8
  • Remark 9