Formal Entropy-Regularized Control of Stochastic Systems

TL;DR

This work obtains bounds on the entropy of system discretizations using traditional formal-abstractions results, and derives an additional bound on the difference between the entropy of a continuous distribution and that of its discretization, which enables formal entropy-aware controller synthesis that trades predictability against control performance while preserving formal guarantees for the original continuous system.

Abstract

Analyzing and controlling system entropy is a powerful tool for regulating predictability of control systems. Applications benefiting from such approaches range from reinforcement learning and data security to human-robot collaboration. In continuous-state stochastic systems, accurate entropy analysis and control remains a challenge. In recent years, finite-state abstractions of continuous systems have enabled control synthesis with formal performance guarantees on objectives such as stage costs. However, these results do not extend to entropy-based performance measures. We solve this problem by first obtaining bounds on the entropy of system discretizations using traditional formal-abstractions results, and then obtaining an additional bound on the difference between the entropy of a continuous distribution and that of its discretization. The resulting theory enables formal entropy-aware controller synthesis that trades predictability against control performance while preserving formal guarantees for the original continuous system. More specifically, we focus on minimizing the linear combination of the KL divergence of the system trajectory distribution to uniform -- our system entropy metric -- and a generic cumulative cost. We note that the bound we derive on the difference between the KL divergence to uniform of a given continuous distribution and its discretization can also be relevant in more general information-theoretic contexts. A set of case studies illustrates the effectiveness of the method.

Figures (5)

• Figure 1: Discretization $(\mathcal{S}_t)_{t\in S}$ of the trajectory space $\mathcal{S}$, as induced by a hyperrectangular discretization $(\mathcal{X}_{i})_{i\in X}$ of the state space $\mathcal{X}$.
• Figure 2: The objects that form a formal lower bound on the KL divergence to uniform of system $\mathcal{M}$, i.e., $\operatorname{KL}(T\| U)$, our metric total system entropy.
• Figure 3: The objects that together form formal upper bounds on the KL divergence to uniform of system $\mathcal{M}$, i.e., $\operatorname{KL}(T\| U)$, our metric total system entropy.
• Figure 4: KL divergence to uniform abstraction bounds as a function of discretization resolution as expressed in the number $N$ of equal subdivisions for each dimension.
• Figure 5: Behavior of the bumpy-hill AV example system under the unregularized policy $\mu_{\text{DP}}$, the entropy-regularized policy with global $\varepsilon$ correction $\mu$, and the locally-corrected entropy-regularized policy $\mu^{\varepsilon}$.

Theorems & Definitions (16)

• Remark 1
• Lemma 1: Continuous-Discrete Discrepancy
• Theorem 1: Entropy Abstraction Lower Bound
• Lemma 2: Discretization Difference Upper-Bound
• Theorem 2: Global Discretization Difference Bound
• Remark 2
• Theorem 3: Local Discretization Difference Bound
• Lemma 3: Bounding the magnitude of the gradient $T$
• Remark 3
• Lemma 4: Derivative of Solution to \ref{['eq:T(c)']}