Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learn to Change the World: Multi-level Reinforcement Learning with Model-Changing Actions

Ziqing Lu, Babak Hassibi, Lifeng Lai, Weiyu Xu

TL;DR

This work considers agents that are not limited to passive adaptations: they instead have model-changing actions that actively modify the RL model of world dynamics itself, and introduces the multi-layer configurable time-varying Markov decision process (MCTVMDP).

Abstract

Reinforcement learning usually assumes a given or sometimes even fixed environment in which an agent seeks an optimal policy to maximize its long-term discounted reward. In contrast, we consider agents that are not limited to passive adaptations: they instead have model-changing actions that actively modify the RL model of world dynamics itself. Reconfiguring the underlying transition processes can potentially increase the agents' rewards. Motivated by this setting, we introduce the multi-layer configurable time-varying Markov decision process (MCTVMDP). In an MCTVMDP, the lower-level MDP has a non-stationary transition function that is configurable through upper-level model-changing actions. The agent's objective consists of two parts: Optimize the configuration policies in the upper-level MDP and optimize the primitive action policies in the lower-level MDP to jointly improve its expected long-term reward.

Learn to Change the World: Multi-level Reinforcement Learning with Model-Changing Actions

TL;DR

This work considers agents that are not limited to passive adaptations: they instead have model-changing actions that actively modify the RL model of world dynamics itself, and introduces the multi-layer configurable time-varying Markov decision process (MCTVMDP).

Abstract

Reinforcement learning usually assumes a given or sometimes even fixed environment in which an agent seeks an optimal policy to maximize its long-term discounted reward. In contrast, we consider agents that are not limited to passive adaptations: they instead have model-changing actions that actively modify the RL model of world dynamics itself. Reconfiguring the underlying transition processes can potentially increase the agents' rewards. Motivated by this setting, we introduce the multi-layer configurable time-varying Markov decision process (MCTVMDP). In an MCTVMDP, the lower-level MDP has a non-stationary transition function that is configurable through upper-level model-changing actions. The agent's objective consists of two parts: Optimize the configuration policies in the upper-level MDP and optimize the primitive action policies in the lower-level MDP to jointly improve its expected long-term reward.
Paper Structure (7 sections, 3 theorems, 37 equations, 3 figures, 1 algorithm)

This paper contains 7 sections, 3 theorems, 37 equations, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem formulation
  3. Cost-constrained optimization problem on TVCMDP
  4. Bi-level Model-based Value Iteration
  5. Performance analysis
  6. Numerical experiments
  7. Jacobian of state-value functions

Key Result

Lemma 1

If $\forall (s,a)\in\mathcal{S}\times\mathcal{A}$, the total variance distance between the empirical kernel $\hat{P}$ and the ground-truth $P$ is bounded by $TV(P(\cdot|s,a), \hat{P}(\cdot|s,a))\leq\delta_g$, then for any policy $\pi:\mathcal{S}\rightarrow\mathcal{A}$, we have where $V_{\max}:=\frac{r_{\max}}{1-\gamma}$. Moreover, let $V^{\pi^*(P)}_{P}$ and $V^{\pi^*(\hat{P})}_{\hat{P}}$ be the c

Figures (3)

  • Figure 1: Bi-level configuration on the synthetic example
  • Figure 2: Improved returns by continuous and bi-level MDP configurations
  • Figure 3: Bi-level Block-world Configuration: Training and test performances

Theorems & Definitions (3)

  • Lemma 1: estimation error Lemma
  • Lemma 2: Propagated Reward Error Bound
  • Lemma 3: Error bound Lemma of Bi-level MDPs