On the hardness of RL with Lookahead

TL;DR

This paper investigates reinforcement learning with transition look-ahead, where the agent is shown the ℓ-step future state tree before acting. It proves a precise complexity boundary: planning with one-step look-ahead (ℓ=1) is solvable in polynomial time via a novel linear programming formulation, for both discounted and average-reward criteria, while planning with two or more steps (ℓ≥2) is NP-hard under both criteria. The authors construct an augmented MDP to encode look-ahead and develop a polynomial-time separation oracle, enabling an ellipsoid-method-based LP solution for ℓ=1. They further show NP-hardness for ℓ≥2 by a reduction from independent set in 3-regular graphs, with a corresponding extension to average reward through a renewal argument. Overall, the work delineates a sharp information-theoretic computational boundary in RL when privileged look-ahead is available, with implications for model-predictive control and prediction-augmented planning.

Abstract

We study reinforcement learning (RL) with transition look-ahead, where the agent may observe which states would be visited upon playing any sequence of $\ell$ actions before deciding its course of action. While such predictive information can drastically improve the achievable performance, we show that using this information optimally comes at a potentially prohibitive computational cost. Specifically, we prove that optimal planning with one-step look-ahead ($\ell=1$) can be solved in polynomial time through a novel linear programming formulation. In contrast, for $\ell \geq 2$, the problem becomes NP-hard. Our results delineate a precise boundary between tractable and intractable cases for the problem of planning with transition look-ahead in reinforcement learning.

Figures (1)

• Figure 1: Hardness of $2-DVDP$ At time $t=0$, the agent is at $s_0$ and observes a subset $Y_0 \subset {\mathcal{S}}_V = \{s_v, v \in V\}$ of reachable states revealed by depth-2 look-ahead. At $s_0$, it may either play $a_1$ to remain in $s_0$ and obtain a new subset $Y_1 \subset$ or choose some $a\neq a_1$ to transition to $s_1$. When in $s_1$ at time $\tau$, the 2-look-ahead removes all remaining randomness in the system, allowing the agent to choose the most rewarding state in $Y_\tau$. Calling $X_v$ the reward of the optimal policy that starts at state $s_v \in {\mathcal{S}}$, an agent at $s_1$ at time $\tau$ can easily reach the expected value $\mathbb{E}[\max_{s \in Y_\tau} X_v]$. Optimizing the time at which an agent must commit to $a_1$ at $s_0$ is then shown to be essentially as hard as finding $\max_{Y \subset [n] : |Y|=k} \mathbb{E}[\max_{i \in Y} X_i]$, which is NP-hard mehta2020hittinghighnotessubset.

Theorems & Definitions (17)

• Definition 1: Push-Forward Operator
• Definition 2: $\ell$-step transition look-ahead
• Remark 1
• Definition 3: Discounted Value Decision Problem ($\ell$-DVDP)
• Definition 4: Average-Reward Decision Problem ($\ell$-ARDP)
• Theorem 1: One-step look-ahead is polynomial-time
• Theorem 2: NP-hardness for $\ell\ge 2$ (discounted)
• Theorem 3: NP-hardness for $\ell \ge 2$ (average reward)
• Lemma S1
• Remark S2