Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Risk-sensitive control as inference with Rényi divergence

Kaito Ito, Kenji Kashima

TL;DR

The risk-sensitive control as inference (RCaI) that extends CaI by using R\'{e}nyi divergence variational inference is introduced and it is shown to be equivalent to log-probability regularized risk-sensitive control, which is an extension of the maximum entropy control.

Abstract

This paper introduces the risk-sensitive control as inference (RCaI) that extends CaI by using Rényi divergence variational inference. RCaI is shown to be equivalent to log-probability regularized risk-sensitive control, which is an extension of the maximum entropy (MaxEnt) control. We also prove that the risk-sensitive optimal policy can be obtained by solving a soft Bellman equation, which reveals several equivalences between RCaI, MaxEnt control, the optimal posterior for CaI, and linearly-solvable control. Moreover, based on RCaI, we derive the risk-sensitive reinforcement learning (RL) methods: the policy gradient and the soft actor-critic. As the risk-sensitivity parameter vanishes, we recover the risk-neutral CaI and RL, which means that RCaI is a unifying framework. Furthermore, we give another risk-sensitive generalization of the MaxEnt control using Rényi entropy regularization. We show that in both of our extensions, the optimal policies have the same structure even though the derivations are very different.

Risk-sensitive control as inference with Rényi divergence

TL;DR

The risk-sensitive control as inference (RCaI) that extends CaI by using R\'{e}nyi divergence variational inference is introduced and it is shown to be equivalent to log-probability regularized risk-sensitive control, which is an extension of the maximum entropy control.

Abstract

This paper introduces the risk-sensitive control as inference (RCaI) that extends CaI by using Rényi divergence variational inference. RCaI is shown to be equivalent to log-probability regularized risk-sensitive control, which is an extension of the maximum entropy (MaxEnt) control. We also prove that the risk-sensitive optimal policy can be obtained by solving a soft Bellman equation, which reveals several equivalences between RCaI, MaxEnt control, the optimal posterior for CaI, and linearly-solvable control. Moreover, based on RCaI, we derive the risk-sensitive reinforcement learning (RL) methods: the policy gradient and the soft actor-critic. As the risk-sensitivity parameter vanishes, we recover the risk-neutral CaI and RL, which means that RCaI is a unifying framework. Furthermore, we give another risk-sensitive generalization of the MaxEnt control using Rényi entropy regularization. We show that in both of our extensions, the optimal policies have the same structure even though the derivations are very different.

Paper Structure

This paper contains 22 sections, 10 theorems, 117 equations, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related work
  4. Notation
  5. Brief introduction to control as inference
  6. Control as Rényi divergence variational inference
  7. Equivalence between CaI with Rényi divergence and risk-sensitive control
  8. Derivation of optimal control and further equivalence results
  9. Another risk-sensitive generalization of MaxEnt control via Rényi entropy
  10. Risk-sensitive reinforcement learning via RCaI
  11. Risk-sensitive policy gradient
  12. Risk-sensitive soft actor-critic
  13. Experiment
  14. Conclusions
  15. More details on Control as Inference
  16. ...and 7 more sections

Key Result

Proposition 1

Assume that $\mu_L ({\mathbb U}) < \infty$ and let $\mathsf{c}_t (x_t,u_t) := c_t (x_t,u_t) + \log \mu_L ({\mathbb U})$. Assume further the existence of density functions $p(x_0)$ and $p(x_{t+1} | x_t, u_t)$ for any $t\in [\![0,T-1]\!]$When considering discrete variables $x_t,u_t$, the assumption $\ where $\diamondsuit$

Figures (7)

  • Figure 1: Relations of control problems.
  • Figure 2: Graphical model for CaI.
  • Figure 3: Average episode cost for RSAC with some $\eta$ and standard SAC.
  • Figure 4: Pendulum length $l = 1.0$ during training
  • Figure 5: System perturbation $l = 1.25$
  • ...and 2 more figures

Theorems & Definitions (12)

  • Proposition 1
  • Theorem 2
  • Theorem 3
  • Corollary 4
  • Lemma 5: Informal
  • Theorem 6
  • Proposition 7
  • Proposition 8
  • Theorem 9
  • proof
  • ...and 2 more