Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning with Function Approximation for Non-Markov Processes

Ali Devran Kara

TL;DR

The paper studies policy evaluation and Q-learning with linear function approximation under non-Markov dynamics and introduces a stationary regime Markov decision process to capture long-run behavior. It proves convergence of policy evaluation updates to the fixed point of the projected Bellman operator $\Pi T^\gamma$ via a Poisson-equation based stochastic approximation, and provides explicit error bounds tied to projection and feature geometry. It shows that Q-learning with linear function approximation generally fails to converge in non-Markov settings, but converges under discretization or certain basis constructions, and extends the framework to POMDPs using finite-memory representations with derived error and near-optimality bounds. Overall, the work connects non-Markov RL to auxiliary Markov models, enabling principled convergence and error analysis for policy evaluation, Q-learning, and finite-memory POMDPs.

Abstract

We study reinforcement learning methods with linear function approximation under non-Markov state and cost processes. We first consider the policy evaluation method and show that the algorithm converges under suitable ergodicity conditions on the underlying non-Markov processes. Furthermore, we show that the limit corresponds to the fixed point of a joint operator composed of an orthogonal projection and the Bellman operator of an auxiliary \emph{Markov} decision process. For Q-learning with linear function approximation, as in the Markov setting, convergence is not guaranteed in general. We show, however, that for the special case where the basis functions are chosen based on quantization maps, the convergence can be shown under similar ergodicity conditions. Finally, we apply our results to partially observed Markov decision processes, where finite-memory variables are used as state representations, and we derive explicit error bounds for the limits of the resulting learning algorithms.

Reinforcement Learning with Function Approximation for Non-Markov Processes

TL;DR

The paper studies policy evaluation and Q-learning with linear function approximation under non-Markov dynamics and introduces a stationary regime Markov decision process to capture long-run behavior. It proves convergence of policy evaluation updates to the fixed point of the projected Bellman operator via a Poisson-equation based stochastic approximation, and provides explicit error bounds tied to projection and feature geometry. It shows that Q-learning with linear function approximation generally fails to converge in non-Markov settings, but converges under discretization or certain basis constructions, and extends the framework to POMDPs using finite-memory representations with derived error and near-optimality bounds. Overall, the work connects non-Markov RL to auxiliary Markov models, enabling principled convergence and error analysis for policy evaluation, Q-learning, and finite-memory POMDPs.

Abstract

We study reinforcement learning methods with linear function approximation under non-Markov state and cost processes. We first consider the policy evaluation method and show that the algorithm converges under suitable ergodicity conditions on the underlying non-Markov processes. Furthermore, we show that the limit corresponds to the fixed point of a joint operator composed of an orthogonal projection and the Bellman operator of an auxiliary \emph{Markov} decision process. For Q-learning with linear function approximation, as in the Markov setting, convergence is not guaranteed in general. We show, however, that for the special case where the basis functions are chosen based on quantization maps, the convergence can be shown under similar ergodicity conditions. Finally, we apply our results to partially observed Markov decision processes, where finite-memory variables are used as state representations, and we derive explicit error bounds for the limits of the resulting learning algorithms.
Paper Structure (21 sections, 17 theorems, 144 equations)

This paper contains 21 sections, 17 theorems, 144 equations.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. Policy Evaluation for Non-Markov Processes
  5. A Stochastic Approximation Result for Non-Markov Processes under Ergodicity
  6. Stationary Regime MDP
  7. Linear Function Approximation and Projection
  8. Convergence of the Policy Evaluation Algorithm
  9. Error Analysis for the Limit Value
  10. On Learning Approximately Optimal Q-Values
  11. Greedy-Policy Covariance Dominance
  12. Convergence under Discretization
  13. Function Approximation for POMDPs using Finite Memory
  14. Partially Observed Markov Decision Processes
  15. Reduction to Fully Observed Using Finite-Memory Variables
  16. ...and 6 more sections

Key Result

Lemma 1

Assumption mixing_assmp implies Assumption poisson (ii). That is, if $\|Y_0^A\|_2 < \infty$, $\|Y_0^b\|_2 < \infty$, and $\sum_{k=0}^\infty \sqrt{\alpha(k)} < \infty$, then the sequences $\{Y^A_t\}$ and $\{Y^b_t\}$ are uniformly bounded for all $t$:

Theorems & Definitions (35)

  • Remark 1
  • Remark 2
  • Lemma 1
  • proof
  • Proposition 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • ...and 25 more