Multi-level meta-reinforcement learning with skill-based curriculum

TL;DR

An efficient multi-level procedure for repeatedly compressing Markov decision processes (MDPs), wherein a parametric family of policies at one level is treated as single actions in the compressed MDPs at higher levels, while preserving the semantic meanings and structure of the original MDP.

Abstract

We consider problems in sequential decision making with natural multi-level structure, where sub-tasks are assembled together to accomplish complex goals. Systematically inferring and leveraging hierarchical structure has remained a longstanding challenge; we describe an efficient multi-level procedure for repeatedly compressing Markov decision processes (MDPs), wherein a parametric family of policies at one level is treated as single actions in the compressed MDPs at higher levels, while preserving the semantic meanings and structure of the original MDP, and mimicking the natural logic to address a complex MDP. Higher-level MDPs are themselves independent MDPs with less stochasticity, and may be solved using existing algorithms. As a byproduct, spatial or temporal scales may be coarsened at higher levels, making it more efficient to find long-term optimal policies. The multi-level representation delivered by this procedure decouples sub-tasks from each other and usually greatly reduces unnecessary stochasticity and the policy search space, leading to fewer iterations and computations when solving the MDPs. A second fundamental aspect of this work is that these multi-level decompositions plus the factorization of policies into embeddings (problem-specific) and skills (including higher-order functions) yield new transfer opportunities of skills across different problems and different levels. This whole process is framed within curriculum learning, wherein a teacher organizes the student agent's learning process in a way that gradually increases the difficulty of tasks and and promotes transfer across MDPs and levels within and across curricula. The consistency of this framework and its benefits can be guaranteed under mild assumptions. We demonstrate abstraction, transferability, and curriculum learning in examples, including MazeBase+, a more complex variant of the MazeBase example.

Figures (11)

• Figure 2: A representation of the second experiment of the MazeBase+ example. The representation in this figure is similar to the one for the first experiment (Fig. ). In the second experiment, we apply our framework to perform efficient transfer to MazeBase+ worlds with different configurations of doors, keys, and the goal.
• Figure 3: Commuting diagram for a skill-embedding decomposition of $\pi_I$.
• Figure 4: We display $\mathbb{E}_{s_0}V_\pi(s_0):=\sum_{s\in{\mathcal{S}}^{\text{init}}\xspace}V(s)/|{\mathcal{S}}^{\text{init}}\xspace|$, the average (over all initial conditions $s_0$) of the $V_{\pi}(s_0)$, where $V_\pi$ is the value function for ${\mathtt{MDP}}_{1,1}$, ${\mathtt{MDP}}_{2,1}$, ${\mathtt{MDP}}_{2,2}$, and ${\mathtt{MDP}}_{3,1}$ respectively, as $\pi$ is optimized during iterations of classical value iteration (in red) and of value iteration within our algorithm, with iterations within ${\mathtt{MDP}}_{1,1}={\mathtt{MDP}}_{\text{dense}}^{\text{nav}}$ in orange, iterations within ${\mathtt{MDP}}_{2,n} (1\leq n\leq 2)$ at level $2$ in blue followed by iterations at level $1$ in orange, and iterations within ${\mathtt{MDP}}_{3,1}$ at level $3$ in green followed by iterations at level $2$ in blue followed by iterations at level $1$ in orange. Although we spend extra effort in solving the ${\mathtt{MDP}}_{2,n}$'s (recall that ${\mathtt{MDP}}_{1,1}$'s comes from the example of navigation and transportation with traffic jams, so we do not spend extra effort there), they prepare us well enough so that we only need a few more iterations for solving the target ${\mathtt{MDP}}_{3,1}$ (green+blue+orange), much fewer than if we solved it from scratch using classical value iteration (red).
• Figure 5: Left: we display $\mathbb{E}_{s_0}V_\pi(s_0)$, where $V_\pi$ is the value function for ${\mathtt{MDP}}_{1,1}$, ${\mathtt{MDP}}'_{2,1}$, ${\mathtt{MDP}}_{2,2}$, and ${\mathtt{MDP}}'_{3,1}$ respectively, as $\pi$ is optimized during iterations of classical value iteration and of value iteration within our algorithm. See Fig. for the detailed explanation, and Fig. for the representation of this second experiment of the MazeBase+ example. One addition here compared with Fig. is that here we also plot iterations within ${\mathtt{MDP}}'_{3,1}$ at level $2$ in purple followed by iterations at level $1$ in yellow, assuming the student did not solve ${\mathtt{MDP}}_{2,2}$ and consequently the assistant did not extract $\overline{\pi}^{\mathrm{concat}}$. This corresponds to treating ${\mathtt{MDP}}'_{3,1}$ as an MDP of difficulty $2$. The extra effort here (purple+yellow) compared with the original case (green+blue+orange) demonstrates the advantage of treating ${\mathtt{MDP}}'_{3,1}$ as an MDP of difficulty $3$ and of extracting the higher-order function $\overline{\pi}^{\mathrm{concat}}$. Right: we display $\mathbb{E}_{s_0}V_\pi(s_0)$, where $V_\pi$ is the value function for ${\mathtt{MDP}}_{1,1}$, ${\mathtt{MDP}}"_{2,1}$, ${\mathtt{MDP}}_{2,2}$, and ${\mathtt{MDP}}"_{3,1}$ respectively, as $\pi$ is optimized during iterations of classical value iteration and of value iteration within our algorithm. See Fig. for the detailed explanation, and see Fig. for the representation of this third experiment of the MazeBase+ example.
• Figure 6: A representation of the third experiment of the MazeBase+ example. The representation in this figure is similar to the one for the first experiment (Fig. ). In the third experiment, we consider the situation where the optimal policy at the highest level, when refined to a finer level, yields a suboptimal policy where the student collects ${}_2$, opens ${\mathtt{door}}_2$, and then collects ${}_3$ (very close to ${}_2$) to then open ${\mathtt{door}}_3$. Within our algorithm, this suboptimal policy gets refined in order to yield the optimal policy, demonstrating the robustness of our optimization procedure; in this case it is also the case that our algorithm still outperforms naïve value iteration (see Fig. (right)).

Theorems & Definitions (17)

• Definition 1
• Definition 2
• Definition 3
• Definition 4
• Definition 5
• Lemma 6
• Corollary 7
• Corollary 8
• Proposition 9
• Theorem 10