Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning

TL;DR

This work tackles policy ambiguity in reinforcement learning by pairing entropy regularization with a vanishing temperature regime to yield a well-defined, interpretable optimal policy. It introduces a temperature decoupling gambit that separates optimization and execution temperatures, enabling convergence of both policies and return distributions to a reference-optimal, diversity-preserving limit $\pi^{\mathsf{ref},\star}$ and $\zeta^{\mathsf{ref},\star}$. The authors develop soft distributional Bellman operators for evaluation and control, proving contraction in distributional metrics and providing convergent iterative schemes to estimate the reference-optimal return distribution in the vanishing-temperature limit. Numerical demonstrations illustrate the distinct limiting behavior of ERL vs. the decoupled scheme and validate the convergence of return distributions under the proposed method. The results lay groundwork for robust, interpretable, distributional RL with provable limits, while highlighting practical limitations and avenues for future algorithmic integration.

Abstract

In the pursuit of finding an optimal policy, reinforcement learning (RL) methods generally ignore the properties of learned policies apart from their expected return. Thus, even when successful, it is difficult to characterize which policies will be learned and what they will do. In this work, we present a theoretical framework for policy optimization that guarantees convergence to a particular optimal policy, via vanishing entropy regularization and a temperature decoupling gambit. Our approach realizes an interpretable, diversity-preserving optimal policy as the regularization temperature vanishes and ensures the convergence of policy derived objects--value functions and return distributions. In a particular instance of our method, for example, the realized policy samples all optimal actions uniformly. Leveraging our temperature decoupling gambit, we present an algorithm that estimates, to arbitrary accuracy, the return distribution associated to its interpretable, diversity-preserving optimal policy.

Figures (5)

• Figure 3.1: Differences between $\hat{\pi}^{\tau,\star}$ and $\hat{\pi}^{\tau,\sigma}$, approximated with soft Q-learning. Left: Graphical model of the MDP; arrow colors encode actions. Center: Depiction of the estimated policies $\hat{\pi}^{\tau,\star}$ at each state, as $\tau\to 0$. Right: Depiction of the estimated policies $\hat{\pi}^{\tau,\sigma}$ at each state, as $\tau\to 0$. Summary: Learned policies differ in $x_0$, but are otherwise the same.
• Figure 4.1: Evolution of the soft optimality iterates $({\mathcal{T}_{\tau}^{\star}})^k\zeta (x, a)$ (bottom row) and the iterates of the distributional optimality operator $({\cal{T}^\star})^{k}{\zeta}(x, a)$ (top row). Video of entire iterate sequence is available at https://harwiltz.github.io/assets/stable-return-distributions/.
• Figure 4.2: An illustrative MDP.
• Figure 4.3: Estimates of return distributions via soft distributional dynamic programming---$\hat{\eta}^{\tau,\sigma}$ using the temperature-decoupling gambit and $\hat{\eta}^{\tau,\star}$ without---as $\tau\to 0$. As the temperature vanishes, $\eta^{\tau,\sigma}$ recovers the return distribution of $\pi^{\mathsf{ref},\star}$, shown on the right.
• Figure 4.4: Return distribution estimation with vanishing temperature using soft distributional dynamic programming, with 32-bit floating point precision.

Theorems & Definitions (96)

• Lemma 2.0
• Definition 2.1
• Theorem 2.2
• Theorem 2.4
• Remark 2.5
• Lemma 3.0
• Theorem 3.1
• Definition 3.2
• Remark 3.4
• Theorem 3.5