Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Linear Bellman Completeness Suffices for Efficient Online Reinforcement Learning with Few Actions

Noah Golowich, Ankur Moitra

TL;DR

The paper addresses online reinforcement learning with linear function approximation under linear Bellman completeness, a setting where statistically efficient algorithms traditionally rely on nonconvex optimism. It introduces PSDP-UCB, a polynomial-time algorithm that achieves near-optimal policies when the number of actions is fixed, by engineering exploration bonuses whose Bellman backups are linear (Bellman-linear). The key contributions include a novel truncation-based bonus construction that preserves Bellman-linearity, a detailed analysis using concentration and elliptical potential tools, and a demonstration that local optimism suffices for computational efficiency in this broad setting. This work bridges the gap between statistical tractability and computational efficiency in RL with linear function approximation and opens avenues for further improvements with larger action sets or weaker completeness assumptions.

Abstract

One of the most natural approaches to reinforcement learning (RL) with function approximation is value iteration, which inductively generates approximations to the optimal value function by solving a sequence of regression problems. To ensure the success of value iteration, it is typically assumed that Bellman completeness holds, which ensures that these regression problems are well-specified. We study the problem of learning an optimal policy under Bellman completeness in the online model of RL with linear function approximation. In the linear setting, while statistically efficient algorithms are known under Bellman completeness (e.g., Jiang et al. (2017); Zanette et al. (2020)), these algorithms all rely on the principle of global optimism which requires solving a nonconvex optimization problem. In particular, it has remained open as to whether computationally efficient algorithms exist. In this paper we give the first polynomial-time algorithm for RL under linear Bellman completeness when the number of actions is any constant.

Linear Bellman Completeness Suffices for Efficient Online Reinforcement Learning with Few Actions

TL;DR

The paper addresses online reinforcement learning with linear function approximation under linear Bellman completeness, a setting where statistically efficient algorithms traditionally rely on nonconvex optimism. It introduces PSDP-UCB, a polynomial-time algorithm that achieves near-optimal policies when the number of actions is fixed, by engineering exploration bonuses whose Bellman backups are linear (Bellman-linear). The key contributions include a novel truncation-based bonus construction that preserves Bellman-linearity, a detailed analysis using concentration and elliptical potential tools, and a demonstration that local optimism suffices for computational efficiency in this broad setting. This work bridges the gap between statistical tractability and computational efficiency in RL with linear function approximation and opens avenues for further improvements with larger action sets or weaker completeness assumptions.

Abstract

One of the most natural approaches to reinforcement learning (RL) with function approximation is value iteration, which inductively generates approximations to the optimal value function by solving a sequence of regression problems. To ensure the success of value iteration, it is typically assumed that Bellman completeness holds, which ensures that these regression problems are well-specified. We study the problem of learning an optimal policy under Bellman completeness in the online model of RL with linear function approximation. In the linear setting, while statistically efficient algorithms are known under Bellman completeness (e.g., Jiang et al. (2017); Zanette et al. (2020)), these algorithms all rely on the principle of global optimism which requires solving a nonconvex optimization problem. In particular, it has remained open as to whether computationally efficient algorithms exist. In this paper we give the first polynomial-time algorithm for RL under linear Bellman completeness when the number of actions is any constant.
Paper Structure (49 sections, 34 theorems, 144 equations, 1 algorithm)

This paper contains 49 sections, 34 theorems, 144 equations, 1 algorithm.

Table of Contents

  1. Introduction
  2. Value iteration and Bellman completeness.
  3. Exploration and linear Bellman completeness
  4. Main result for the online setting
  5. Proof idea of \ref{['thm:online-intro']}.
  6. Organization of the paper.
  7. Preliminaries
  8. The online learning problem
  9. Function approximation
  10. Polytope of actions.
  11. Gaussian smoothing.
  12. Miscellaneous notation.
  13. Linear policies
  14. Technical overview
  15. Prior work: exploration via optimism
  16. ...and 34 more sections

Key Result

Theorem 1.1

Suppose the ground-truth MDP satisfies linear Bellman completeness. Then for any $\epsilon > 0$, there is an algorithm (PSDP-UCB; alg:psdp-ucb) which with high probability learns an $\epsilon$-optimal policy using $(Hd|\mathcal{A}|\epsilon^{-1})^{O(|\mathcal{A}|)}$ samples and time.

Theorems & Definitions (66)

  • Theorem 1.1: Informal version of \ref{['thm:policy-learning']}
  • Definition 2.1: Linear Bellman Completeness
  • Lemma 3.1: Informal version of \ref{['lem:approx-optimism']}
  • Lemma 3.2: Informal version of \ref{['lem:sigmap-bound', 'lem:quadratic-sim', 'lem:orig-truncated']}
  • Definition 3.1
  • Lemma 3.3: Variant of \ref{['lem:wht-exist-bound']}
  • Lemma 3.4: Simplified version of \ref{['lem:quadratic-sim']}
  • Lemma 4.1: Limiting preserves Bellman-linearity
  • proof
  • Lemma 4.2: Differentiation preserves Bellman-linearity
  • ...and 56 more