Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Finite-Time Bounds for Average-Reward Fitted Q-Iteration

Jongmin Lee, Ernest K. Ryu

TL;DR

The paper tackles the problem of sample-efficient offline RL under the average-reward criterion with general function approximation, focusing on weakly communicating MDPs.It introduces Anchored Fitted Q-Iteration (Anc-F-QI), which couples standard FQI with an anchor to enable finite-time analysis, and establishes IID and single-trajectory data results with concrete sample complexity bounds.To further improve efficiency, it develops Relative Anchored FQI (R-Anc-F-QI), achieving refined rates of $\tilde{O}(1/\\epsilon^4)$ for IID and $\tilde{O}(1/\\epsilon^8)$ for single-trajectory data, under the relative normalization framework.The work highlights that a full-coverage assumption under Bellman-optimality is central to the analysis, discusses limitations, and outlines future work such as relaxing coverage requirements and incorporating variance-reduction ideas.

Abstract

Although there is an extensive body of work characterizing the sample complexity of discounted-return offline RL with function approximations, prior work on the average-reward setting has received significantly less attention, and existing approaches rely on restrictive assumptions, such as ergodicity or linearity of the MDP. In this work, we establish the first sample complexity results for average-reward offline RL with function approximation for weakly communicating MDPs, a much milder assumption. To this end, we introduce Anchored Fitted Q-Iteration, which combines the standard Fitted Q-Iteration with an anchor mechanism. We show that the anchor, which can be interpreted as a form of weight decay, is crucial for enabling finite-time analysis in the average-reward setting. We also extend our finite-time analysis to the setup where the dataset is generated from a single-trajectory rather than IID transitions, again leveraging the anchor mechanism.

Finite-Time Bounds for Average-Reward Fitted Q-Iteration

TL;DR

The paper tackles the problem of sample-efficient offline RL under the average-reward criterion with general function approximation, focusing on weakly communicating MDPs.It introduces Anchored Fitted Q-Iteration (Anc-F-QI), which couples standard FQI with an anchor to enable finite-time analysis, and establishes IID and single-trajectory data results with concrete sample complexity bounds.To further improve efficiency, it develops Relative Anchored FQI (R-Anc-F-QI), achieving refined rates of $\tilde{O}(1/\\epsilon^4)$ for IID and $\tilde{O}(1/\\epsilon^8)$ for single-trajectory data, under the relative normalization framework.The work highlights that a full-coverage assumption under Bellman-optimality is central to the analysis, discusses limitations, and outlines future work such as relaxing coverage requirements and incorporating variance-reduction ideas.

Abstract

Although there is an extensive body of work characterizing the sample complexity of discounted-return offline RL with function approximations, prior work on the average-reward setting has received significantly less attention, and existing approaches rely on restrictive assumptions, such as ergodicity or linearity of the MDP. In this work, we establish the first sample complexity results for average-reward offline RL with function approximation for weakly communicating MDPs, a much milder assumption. To this end, we introduce Anchored Fitted Q-Iteration, which combines the standard Fitted Q-Iteration with an anchor mechanism. We show that the anchor, which can be interpreted as a form of weight decay, is crucial for enabling finite-time analysis in the average-reward setting. We also extend our finite-time analysis to the setup where the dataset is generated from a single-trajectory rather than IID transitions, again leveraging the anchor mechanism.
Paper Structure (36 sections, 16 theorems, 97 equations, 1 figure, 1 table, 2 algorithms)

This paper contains 36 sections, 16 theorems, 97 equations, 1 figure, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contribution.
  3. Preliminaries and notations
  4. Average-reward MDP.
  5. Coverage coefficient.
  6. Value Iteration.
  7. MDP classes.
  8. Conditions of Prior works
  9. Uniform mixing ozdaglar2024.
  10. All-policy Bellman equation and linear MDPs gabbianelli2024offline.
  11. Bellman optimality equation (our work).
  12. Anchored Fitted Q-Iteration
  13. The anchor mechanism and weight decay
  14. Assumptions on the function space $\mathcal{F}$
  15. Approximate Anchored Q-Value Iteration
  16. ...and 21 more sections

Key Result

Proposition 1

Let $p\in [1,\infty]$, and let $\mu$ and $\rho$ be distributions on $\mathcal{S} \times \mathcal{A}$. Under Assumption assum_bell_opt and converage_tran (Bellman optimality equation, uniform stochastic transition), the policy error of eq:Apx-Anc-QI with $\lambda_k = \frac{k}{k+2}$ satisfies Similarly, under Assumption assum_bell_opt and converage_fut (Bellman optimality equation, uniform future s

Figures (1)

  • Figure 1: The MDP classes satisfy the inclusion: ergodic $\subset$ unichain $\subset$ weakly communicating

Theorems & Definitions (36)

  • Definition 1: Inherent Bellman error
  • Proposition 1
  • Definition 2
  • Lemma 1
  • Theorem 1
  • Definition 3: $\beta$-mixing
  • Definition 4: pseudo dimension
  • Lemma 2
  • Theorem 2
  • Theorem 3
  • ...and 26 more