Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Computational Hardness of Reinforcement Learning with Partial $q^π$-Realizability

Shayan Karimi, Xiaoqi Tan

TL;DR

The work analyzes computational hardness for reinforcement learning under partial $q^{\pi}$-realizability, a regime that lies between $q^*$-realizability and $q^{\pi}$-realizability. It defines two hardness settings—greedy-policy (NP-hard) and softmax-policy (hard under rETH)—and proves these via reductions from $\delta$-Max-3SAT and related variants, showing hardness persists even as the policy class grows beyond the optimal policy. The key technique is constructing polynomial-time MDP reductions that encode SAT instances into linear realizability structures, with explicit feature and weight designs for both greedy and softmax policies. The findings highlight intrinsic computational barriers for RL with linear function approximation in a broad regime and motivate further exploration under additional structural assumptions, such as agnostic RL settings.

Abstract

This paper investigates the computational complexity of reinforcement learning in a novel linear function approximation regime, termed partial $q^π$-realizability. In this framework, the objective is to learn an $ε$-optimal policy with respect to a predefined policy set $Π$, under the assumption that all value functions for policies in $Π$ are linearly realizable. The assumptions of this framework are weaker than those in $q^π$-realizability but stronger than those in $q^*$-realizability, providing a practical model where function approximation naturally arises. We prove that learning an $ε$-optimal policy in this setting is computationally hard. Specifically, we establish NP-hardness under a parameterized greedy policy set (argmax) and show that - unless NP = RP - an exponential lower bound (in feature vector dimension) holds when the policy set contains softmax policies, under the Randomized Exponential Time Hypothesis. Our hardness results mirror those in $q^*$-realizability and suggest computational difficulty persists even when $Π$ is expanded beyond the optimal policy. To establish this, we reduce from two complexity problems, $δ$-Max-3SAT and $δ$-Max-3SAT(b), to instances of GLinear-$κ$-RL (greedy policy) and SLinear-$κ$-RL (softmax policy). Our findings indicate that positive computational results are generally unattainable in partial $q^π$-realizability, in contrast to $q^π$-realizability under a generative access model.

Computational Hardness of Reinforcement Learning with Partial $q^π$-Realizability

TL;DR

The work analyzes computational hardness for reinforcement learning under partial -realizability, a regime that lies between -realizability and -realizability. It defines two hardness settings—greedy-policy (NP-hard) and softmax-policy (hard under rETH)—and proves these via reductions from -Max-3SAT and related variants, showing hardness persists even as the policy class grows beyond the optimal policy. The key technique is constructing polynomial-time MDP reductions that encode SAT instances into linear realizability structures, with explicit feature and weight designs for both greedy and softmax policies. The findings highlight intrinsic computational barriers for RL with linear function approximation in a broad regime and motivate further exploration under additional structural assumptions, such as agnostic RL settings.

Abstract

This paper investigates the computational complexity of reinforcement learning in a novel linear function approximation regime, termed partial -realizability. In this framework, the objective is to learn an -optimal policy with respect to a predefined policy set , under the assumption that all value functions for policies in are linearly realizable. The assumptions of this framework are weaker than those in -realizability but stronger than those in -realizability, providing a practical model where function approximation naturally arises. We prove that learning an -optimal policy in this setting is computationally hard. Specifically, we establish NP-hardness under a parameterized greedy policy set (argmax) and show that - unless NP = RP - an exponential lower bound (in feature vector dimension) holds when the policy set contains softmax policies, under the Randomized Exponential Time Hypothesis. Our hardness results mirror those in -realizability and suggest computational difficulty persists even when is expanded beyond the optimal policy. To establish this, we reduce from two complexity problems, -Max-3SAT and -Max-3SAT(b), to instances of GLinear--RL (greedy policy) and SLinear--RL (softmax policy). Our findings indicate that positive computational results are generally unattainable in partial -realizability, in contrast to -realizability under a generative access model.
Paper Structure (21 sections, 8 theorems, 55 equations, 1 figure, 4 tables)

This paper contains 21 sections, 8 theorems, 55 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Problem Statement and Main Results
  4. Problem Statement and Definitions
  5. Main Results: Hardness of Solving gLinear-$\kappa$-RL and sLinear-$\kappa$-RL
  6. Proof Sketch of Theorem \ref{['main-thm']}
  7. An Overview of Our Proof
  8. An In-depth Look at Step-1: Polynomial Construction of $M_{\varphi}$
  9. Design of the MDP Structure
  10. Design of PSP Feature and Weight Vectors $\phi'$ and $\theta'$
  11. Design of partial realizability Feature and Weight Vectors $\phi$ and $\theta$
  12. Conclusion and Future Work
  13. Extended Discussion of Related Work
  14. An Example of the MDP Instance $M_{\varphi}$
  15. Remaining Proof of Proposition \ref{['propos : realizability']}
  16. ...and 6 more sections

Key Result

Theorem 3.1

Solving the gLinear-$\kappa$-RL problem for $\epsilon \leq 0.05$ is NP-hard. More specifically, there exists no polynomial-time algorithm (in terms of the parameters $(d, d', H)$) to compute an $\epsilon$-optimal policy for the gLinear-$\kappa$-RL problem, unless $\text{P = NP}$.

Figures (1)

  • Figure 4.2.1: Given a Max-3SAT problem $\varphi: (x_1 \vee \bar{x}_2 \vee x_3)\wedge (\bar{x}_1 \vee x_2 \vee \bar{x}_3)$ with $n = 3$ variables, the constructed MDP consists of 15 states, each represented as a $n$-tuple of -1, 0, and 1. The initial state is defined as $(-1, -1, -1)$ (i.e., the root of the tree). Actions are either “True” or “False,” and the total horizon is $H = n + 1 = 4$. Starting from the initial state $(-1, -1, -1)$ at step $h = 1$, if the action “False” is taken (i.e., $a_1 = 0$), the first element of the next state is set to $0$, resulting in the state $(0, -1, -1)$ at step $h = 2$ (the red path). Alternatively, if the action “True” is taken (i.e., $a_1 = 1$), the first element of the next state is set to $1$, leading to the state $(1, -1, -1)$ at step $h = 2$ (the blue path). In the figure, both the red and blue paths yield a total reward of $\frac{1}{2}$.

Theorems & Definitions (26)

  • Definition 2.1: Randomized Exponential Time Hypothesis (rETH) Dell_2014
  • Definition 3.1: Partial $q^{\pi}$-realizability under $\Pi$
  • Definition 3.2: Learner's objective: $\epsilon$-optimality relative to $\Pi$
  • Definition 3.3: Greedy policy set $\Pi^{g}$
  • Definition 3.3: Greedy policy set $\Pi^{g}$
  • Definition 3.4: Softmax Policy Set $\Pi^{sm}$
  • Theorem 3.1: NP-Hardness of gLinear-$\kappa$-RL
  • Theorem 3.2: Hardness under $\Pi^{sm}$
  • Definition 4.1: Max-3SAT
  • Definition 4.2: $\delta$-Max-3SAT
  • ...and 16 more