Generalized Maximum Entropy Differential Dynamic Programming

TL;DR

A sampling-based trajectory optimization method derived from the maximum entropy formulation of Differential Dynamic Programming with Tsallis entropy, which leads to a Gaussian optimal control policy for exploration during optimization, in the form of q-Gaussian.

Abstract

We present a sampling-based trajectory optimization method derived from the maximum entropy formulation of Differential Dynamic Programming with Tsallis entropy. This method is a generalization of the legacy work with Shannon entropy, which leads to a Gaussian optimal control policy for exploration during optimization. With the Tsallis entropy, the policy takes the form of $q$-Gaussian, which further encourages exploration with its heavy-tailed shape. Moreover, the sampling variance is scaled according to the value function of the trajectory. This scaling mechanism is the unique property of the algorithm with Tsallis entropy in contrast to the original formulation with Shannon entropy, which scales variance with a fixed temperature parameter. Due to this property, our proposed algorithms can promote exploration when necessary, that is, the cost of the trajectory is high. The simulation results with two robotic systems with multimodal cost demonstrate the properties of the proposed algorithm.

Figures (4)

• Figure 1: $q$-Gaussian distribution with different $q$s with $\mu_{q}=0$ and $\sigma_{q}^{2}=1$. $q=1$ corresponds to a normal Gaussian.
• Figure 2: Normal Gaussian policy, $q$-Gaussian policy $\pi$ and $q$-escort distribution of $\pi^{q}/C$ policy with different value function.
• Figure 3: Comparison of normal DDP, multimodal ME-DDP with Shannon entropy, and ME-DDP with Tsallis entropy with two different $\alpha$s. Trajectories of the 15 experiments are overlaid. $\times$ and $\bigstar$ indicate the initial and target positions, respectively. The obstacles are drawn in gray. In the quadrotor example, obstacles are drawn in transparent.
• Figure 4: Evolution of mean cost over iterations. The numbers in the legends corresponds to those in Fig.\ref{['fig:comparison']}