Beyond KL-divergence: Risk Aware Control Through Cross Entropy and Adversarial Entropy Regularization
Menno van Zutphen, Domagoj Herceg, Duarte J. Antunes
TL;DR
The paper tackles robust control under distributional uncertainty by introducing a risk-aware regularization that combines a cross-entropy term $H_c(p,r)$ and an entropy term $H(p)$ with weights $\gamma_H$ and $\gamma_E$. This yields a DP-like minsoftmax framework in which the adversary's distribution has a closed-form softmax form $p_k^{*}(w|x,u) \propto e^{\alpha_k/\gamma_E}$ with $\alpha_k=\gamma_H\log r(w|x,u) + J_{k+1}(f(x,u,w))$, enabling efficient computation of the optimal policy. In the linear-quadratic Gaussian setting, the method recovers $\mathcal{H}_\infty$-control in the infinite-horizon limit and can interpolate among ML-Certainty Equivalence, KL-regularization, and risk-sensitive controllers via parameter tuning. The approach is demonstrated on a numerical irrigation example, showing how parameter choices trade off conservatism against empirical disturbance information. Overall, minsoftmax provides a flexible, analytically tractable framework that unifies several robust control paradigms for practical, uncertainty-aware decision making.
Abstract
While the idea of robust dynamic programming (DP) is compelling for systems affected by uncertainty, addressing worst-case disturbances generally results in excessive conservatism. This paper introduces a method for constructing control policies robust to adversarial disturbance distributions that relate to a provided empirical distribution. The character of the adversary is shaped by a regularization term comprising a weighted sum of (i) the cross-entropy between the empirical and the adversarial distributions, and (ii) the entropy of the adversarial distribution itself. The regularization weights are interpreted as the likelihood factor and the temperature respectively. The proposed framework leads to an efficient DP-like algorithm -- referred to as the minsoftmax algorithm -- to obtain the optimal control policy, where the disturbances follow an analytical softmax distribution in terms of the empirical distribution, temperature, and likelihood factor. It admits a number of control-theoretic interpretations and can thus be understood as a flexible tool for integrating complementary features of related control frameworks. In particular, in the linear model quadratic cost setting, with a Gaussian empirical distribution, we draw connections to the well-known $\mathcal{H}_{\infty}$-control. We illustrate our results through a numerical example.