Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Misspecified $Q$-Learning with Sparse Linear Function Approximation: Tight Bounds on Approximation Error

Ally Yalei Du, Lin F. Yang, Ruosong Wang

TL;DR

This work analyzes reinforcement learning with misspecified, sparse linear Q-functions, showing that naïve Bellman-backup or OLIVE-style methods can be suboptimal in this setting. It introduces a novel elimination-based algorithm that maintains per-level candidate parameter sets and uses empirical Bellman errors to prune, achieving a policy within $O(Hε)$ of optimal with polynomial sample complexity in the feature dimension $d$ and horizon $H$ when the sparsity $k$ is constant. The authors establish information-theoretic lower bounds of $ ilde{Ω}(Hε)$, demonstrating the near-optimality of their upper bound and clarifying the fundamental limits of this Misspecified RL setting. Together, the results delineate a tractable regime for misspecified RL with sparse linear approximations and highlight the crucial role of level-wise structure and sparsity in achieving efficient learning.

Abstract

The recent work by Dong & Yang (2023) showed for misspecified sparse linear bandits, one can obtain an $O\left(ε\right)$-optimal policy using a polynomial number of samples when the sparsity is a constant, where $ε$ is the misspecification error. This result is in sharp contrast to misspecified linear bandits without sparsity, which require an exponential number of samples to get the same guarantee. In order to study whether the analog result is possible in the reinforcement learning setting, we consider the following problem: assuming the optimal $Q$-function is a $d$-dimensional linear function with sparsity $k$ and misspecification error $ε$, whether we can obtain an $O\left(ε\right)$-optimal policy using number of samples polynomially in the feature dimension $d$. We first demonstrate why the standard approach based on Bellman backup or the existing optimistic value function elimination approach such as OLIVE (Jiang et al., 2017) achieves suboptimal guarantees for this problem. We then design a novel elimination-based algorithm to show one can obtain an $O\left(Hε\right)$-optimal policy with sample complexity polynomially in the feature dimension $d$ and planning horizon $H$. Lastly, we complement our upper bound with an $\widetildeΩ\left(Hε\right)$ suboptimality lower bound, giving a complete picture of this problem.

Misspecified $Q$-Learning with Sparse Linear Function Approximation: Tight Bounds on Approximation Error

TL;DR

This work analyzes reinforcement learning with misspecified, sparse linear Q-functions, showing that naïve Bellman-backup or OLIVE-style methods can be suboptimal in this setting. It introduces a novel elimination-based algorithm that maintains per-level candidate parameter sets and uses empirical Bellman errors to prune, achieving a policy within of optimal with polynomial sample complexity in the feature dimension and horizon when the sparsity is constant. The authors establish information-theoretic lower bounds of , demonstrating the near-optimality of their upper bound and clarifying the fundamental limits of this Misspecified RL setting. Together, the results delineate a tractable regime for misspecified RL with sparse linear approximations and highlight the crucial role of level-wise structure and sparsity in achieving efficient learning.

Abstract

The recent work by Dong & Yang (2023) showed for misspecified sparse linear bandits, one can obtain an -optimal policy using a polynomial number of samples when the sparsity is a constant, where is the misspecification error. This result is in sharp contrast to misspecified linear bandits without sparsity, which require an exponential number of samples to get the same guarantee. In order to study whether the analog result is possible in the reinforcement learning setting, we consider the following problem: assuming the optimal -function is a -dimensional linear function with sparsity and misspecification error , whether we can obtain an -optimal policy using number of samples polynomially in the feature dimension . We first demonstrate why the standard approach based on Bellman backup or the existing optimistic value function elimination approach such as OLIVE (Jiang et al., 2017) achieves suboptimal guarantees for this problem. We then design a novel elimination-based algorithm to show one can obtain an -optimal policy with sample complexity polynomially in the feature dimension and planning horizon . Lastly, we complement our upper bound with an suboptimality lower bound, giving a complete picture of this problem.
Paper Structure (36 sections, 13 theorems, 35 equations, 2 figures)

This paper contains 36 sections, 13 theorems, 35 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Our Contributions
  3. Our Assumption.
  4. The Level-by-level Approach.
  5. Optimistic Value Function Elimination.
  6. The Hardness Result.
  7. Related Work
  8. Preliminaries
  9. Reinforcement Learning
  10. Interacting with an MDP.
  11. The Bandits Setting.
  12. Hardness Results
  13. Warmup: Hardness Result for RL without Samples
  14. The Hard Instance.
  15. The Lower Bound.
  16. ...and 21 more sections

Key Result

Theorem 1.2

Under Assumption assump:approx with $d=k=1$, any bandit algorithm that returns an estimate $\hat{r}$ such that $|\hat{r}(a) - r(a) | < 2\epsilon$ for all arms $a$ with probability at least $0.95$ requires at least $0.9n$ samples, where $n$ is the total number of arms.

Figures (2)

  • Figure 1: Illustration of the hard instance for Theorem \ref{['thm:lb_no_sample']}.
  • Figure 2: Illustration of the hard instance for Theorem \ref{['thm:lb_sample']}.

Theorems & Definitions (31)

  • Theorem 1.2
  • Proposition 1.3
  • Theorem 1.4
  • Theorem 3.1
  • Theorem 3.2
  • Definition 3.3
  • Definition 3.4
  • Lemma 3.5
  • Definition 4.1: Average Bellman error
  • Lemma 4.2
  • ...and 21 more