Leveraging Randomized Smoothing for Optimal Control of Nonsmooth Dynamical Systems

TL;DR

The paper addresses the challenge of applying gradient-based optimal control to nonsmooth dynamics by introducing randomized smoothing (RS) to create differentiable approximations of the dynamics. It then embeds RS into Differential Dynamic Programming to form Randomized DDP (R-DDP), with an adaptive scheme that progressively reduces smoothing noise. By linking RS to RL exploration, the authors provide a unified view of how stochasticity can reveal informative gradients while preserving OC structure. Experiments across pendulum, cube manipulation, quadrotor takeoff, and Solo hopping demonstrate that R-DDP solves non-smooth dynamics where classical OC struggles and RL can be sample-inefficient, highlighting the practical impact for efficient robotic control under friction and contact.

Abstract

Optimal control (OC) algorithms such as Differential Dynamic Programming (DDP) take advantage of the derivatives of the dynamics to efficiently control physical systems. Yet, in the presence of nonsmooth dynamical systems, such class of algorithms are likely to fail due, for instance, to the presence of discontinuities in the dynamics derivatives or because of non-informative gradient. On the contrary, reinforcement learning (RL) algorithms have shown better empirical results in scenarios exhibiting non-smooth effects (contacts, frictions, etc). Our approach leverages recent works on randomized smoothing (RS) to tackle non-smoothness issues commonly encountered in optimal control, and provides key insights on the interplay between RL and OC through the prism of RS methods. This naturally leads us to introduce the randomized Differential Dynamic Programming (R-DDP) algorithm accounting for deterministic but non-smooth dynamics in a very sample-efficient way. The experiments demonstrate that our method is able to solve classic robotic problems with dry friction and frictional contacts, where classical OC algorithms are likely to fail and RL algorithms require in practice a prohibitive number of samples to find an optimal solution.

Figures (8)

• Figure 1: Illustration of randomized smoothing effects on the front left leg of the Solo robot.
• Figure 2: Top: A slight vertical force cannot break the unilateral contact, leaving the cube on the floor and, thus, the state unchanged. Bottom: The non-smoothness of physics induces null gradients $\nabla_{\boldsymbol{u}} J$ which results in the failure of classical optimization techniques (\ref{['sec:local_OC']}).
• Figure 3: Left: The randomly-smoothed dynamics allow to get non-null gradients and escape the local maximum for the inverted pendulum. Right:$Q_0(x,\cdot)$ value function of the pendulum with dry friction exhibits plateaus where $Q_u=0$ around $u=0$ while randomly smoothed dynamics leads to $Q_u \neq 0$.
• Figure 4: Randomized smoothing allows to escape from the local optima $(x,u)=0$ for the inverted pendulum (see Fig \ref{['fig:Qfunc_pendulum']}) and the system reaches the upward position (orange). Without randomized smoothing, the gradients are null and the system remains stuck in the downward position (blue).
• Figure 5: Top: Application of RDDP to precisely control a cartpole movement requiring overcoming dry friction on the joints and moving the pole straight upward. Bottom: Using RDDP for the cartpole task, we compare the influence of the sample size on the solution. Only a few samples (here M=8) are necessary to get reasonable results. Increasing the number of samples in the first order Monte-Carlo estimator of the gradients leads to very marginal improvement.