On the Convergence and Stability of Upside-Down Reinforcement Learning, Goal-Conditioned Supervised Learning, and Online Decision Transformers

TL;DR

This work initiates a theoretical foundation for algorithms that build on the broad paradigm of approaching reinforcement learning through supervised learning or sequence modeling, by providing a rigorous analysis of convergence and stability of Episodic Upside-Down Reinforcement Learning, Goal-Conditioned Supervised Learning and Online Decision Transformers.

Abstract

This article provides a rigorous analysis of convergence and stability of Episodic Upside-Down Reinforcement Learning, Goal-Conditioned Supervised Learning and Online Decision Transformers. These algorithms performed competitively across various benchmarks, from games to robotic tasks, but their theoretical understanding is limited to specific environmental conditions. This work initiates a theoretical foundation for algorithms that build on the broad paradigm of approaching reinforcement learning through supervised learning or sequence modeling. At the core of this investigation lies the analysis of conditions on the underlying environment, under which the algorithms can identify optimal solutions. We also assess whether emerging solutions remain stable in situations where the environment is subject to tiny levels of noise. Specifically, we study the continuity and asymptotic convergence of command-conditioned policies, values and the goal-reaching objective depending on the transition kernel of the underlying Markov Decision Process. We demonstrate that near-optimal behavior is achieved if the transition kernel is located in a sufficiently small neighborhood of a deterministic kernel. The mentioned quantities are continuous (with respect to a specific topology) at deterministic kernels, both asymptotically and after a finite number of learning cycles. The developed methods allow us to present the first explicit estimates on the convergence and stability of policies and values in terms of the underlying transition kernels. On the theoretical side we introduce a number of new concepts to reinforcement learning, like working in segment spaces, studying continuity in quotient topologies and the application of the fixed-point theory of dynamical systems. The theoretical study is accompanied by a detailed investigation of example environments and numerical experiments.

Figures (14)

• Figure 1: Illustration of discontinuity of eUDRL-generated goal-reaching objective along two continuous one-parameter rays (items $A$ and $C$) of environments (transition kernels). Horizontal axes show the value of the ray-parameter $\alpha$; the point of intersection is $\alpha=0$. The exact value of the respective quantities at $\alpha=0$ is represented by a horizontal line (item $B$).
• Figure 2: Illustration of discontinuity of eUDRL-generated quantities at the boundary of $(\Delta \mathcal{S})^{\mathcal{S}\times\mathcal{A}}$. The figures present values of policy and goal-reaching objective along two continuous one-parameter rays in $(\Delta \mathcal{S})^{\mathcal{S}\times\mathcal{A}}$, see items $A$ and $C$ in the legend. Horizontal axes show the value of the ray-parameter $\alpha$; the boundary is reached at $\alpha=0$, where the rays intersect. The exact value of the respective quantities at the boundary is represented by a horizontal line, see item $B$.
• Figure 3: Illustration of the behavior of $\min_{\bar{s}\in\bar{\mathcal{S}}_{\mathit{\lambda}_0}}\pi_n(\mathcal{O}(\bar{s})|\bar{s})$ when varying the distance to a deterministic kernel and varying the initial policy for a random walk on $\mathbb{Z}^3$. The legend shows the distance $\delta$ of the respective graph to the deterministic kernel. Each initial condition is depicted by a different color and plotted for various distances $\delta$. One particular initial condition is highlighted in black.
• Figure 4: Illustration of the continuity of the goal-reaching objective for varying distance to a deterministic kernel and varying initial policy for a random walk on $\mathbb{Z}^3$. The legend shows the distance $\delta$ of the respective graph to the deterministic kernel. Each initial condition is depicted by a different color and plotted for various distances $\delta$. One particular initial condition is highlighted in black in (a). One particular example is highlighted in closer zoom in (b).
• Figure 5: Illustration of the dynamical system induced by iterative application of $f_{\gamma}$. The figure shows the dependence of $f_{\gamma}^{\circ n}(x)$ given initial condition $x$ on iteration $n$.

Theorems & Definitions (58)

• Definition 1: Command Extension
• Remark 2: Absorbing State of CE
• Lemma 3
• Definition 4
• Definition 5
• Lemma 6
• Lemma 7
• Definition 8
• Lemma 9
• Lemma 10