Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stabilizing Fixed-Point Iteration for Markov Chain Poisson Equations

Yang Xu, Vaneet Aggarwal

TL;DR

This work tackles the instability and non-uniqueness of Poisson equations for finite-state Markov chains beyond ergodicity, where unit-modulus eigenvalues prevent standard fixed-point convergence. By removing all non-decaying directions via a real peripheral subspace $\\mathcal{K}(P)$ and solving on the quotient, the authors obtain a contractive operator with a unique quotient solution $[v^\\star]$, along with a gauge-fixed representative $v^\\star$ and a peripheral residual $g^\\star\\in\\mathcal{K}(P)$. They develop an end-to-end pipeline that learns the chain structure, constructs an anchor-based gauge map, and performs projected stochastic approximation to estimate the quotient solution and residuals, achieving a convergence rate of $\\widetilde{O}(T^{-1/2})$ up to estimation error. The method handles multichain and periodic regimes, enabling stable Poisson equation learning for average-reward reinforcement learning beyond ergodicity with interpretable decompositions into recurrent-class/phase structure and transient costs. The approach avoids explicitly estimating the transition matrix, instead exploiting structural properties and anchors to produce scalable, sample-efficient performance evaluation primitives for non-ergodic settings.

Abstract

Poisson equations underpin average-reward reinforcement learning, but beyond ergodicity they can be ill-posed, meaning that solutions are non-unique and standard fixed point iterations can oscillate on reducible or periodic chains. We study finite-state Markov chains with $n$ states and transition matrix $P$. We show that all non-decaying modes are captured by a real peripheral invariant subspace $\mathcal{K}(P)$, and that the induced operator on the quotient space $\mathbb{R}^n/\mathcal{K}(P)$ is strictly contractive, yielding a unique quotient solution. Building on this viewpoint, we develop an end-to-end pipeline that learns the chain structure, estimates an anchor based gauge map, and runs projected stochastic approximation to estimate a gauge-fixed representative together with an associated peripheral residual. We prove $\widetilde{O}(T^{-1/2})$ convergence up to projection estimation error, enabling stable Poisson equation learning for multichain and periodic regimes with applications to performance evaluation of average-reward reinforcement learning beyond ergodicity.

Stabilizing Fixed-Point Iteration for Markov Chain Poisson Equations

TL;DR

This work tackles the instability and non-uniqueness of Poisson equations for finite-state Markov chains beyond ergodicity, where unit-modulus eigenvalues prevent standard fixed-point convergence. By removing all non-decaying directions via a real peripheral subspace and solving on the quotient, the authors obtain a contractive operator with a unique quotient solution , along with a gauge-fixed representative and a peripheral residual . They develop an end-to-end pipeline that learns the chain structure, constructs an anchor-based gauge map, and performs projected stochastic approximation to estimate the quotient solution and residuals, achieving a convergence rate of up to estimation error. The method handles multichain and periodic regimes, enabling stable Poisson equation learning for average-reward reinforcement learning beyond ergodicity with interpretable decompositions into recurrent-class/phase structure and transient costs. The approach avoids explicitly estimating the transition matrix, instead exploiting structural properties and anchors to produce scalable, sample-efficient performance evaluation primitives for non-ergodic settings.

Abstract

Poisson equations underpin average-reward reinforcement learning, but beyond ergodicity they can be ill-posed, meaning that solutions are non-unique and standard fixed point iterations can oscillate on reducible or periodic chains. We study finite-state Markov chains with states and transition matrix . We show that all non-decaying modes are captured by a real peripheral invariant subspace , and that the induced operator on the quotient space is strictly contractive, yielding a unique quotient solution. Building on this viewpoint, we develop an end-to-end pipeline that learns the chain structure, estimates an anchor based gauge map, and runs projected stochastic approximation to estimate a gauge-fixed representative together with an associated peripheral residual. We prove convergence up to projection estimation error, enabling stable Poisson equation learning for multichain and periodic regimes with applications to performance evaluation of average-reward reinforcement learning beyond ergodicity.
Paper Structure (46 sections, 32 theorems, 229 equations, 1 figure, 1 table, 5 algorithms)

This paper contains 46 sections, 32 theorems, 229 equations, 1 figure, 1 table, 5 algorithms.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related work
  4. Average-reward MDP beyond ergodicity.
  5. Ergodic average-reward MDP TD learning.
  6. Poisson equations for Markov chains.
  7. Problem setup
  8. Poisson equation beyond ergodicity
  9. Quotient contraction framework
  10. Peripheral invariant subspace
  11. Quotient space and induced operator
  12. Gauge projection and a generalized Poisson target
  13. Implications on performance evaluation
  14. Algorithm
  15. Learning the gauge map
  16. ...and 31 more sections

Key Result

Theorem 1

Let $Q \coloneqq \mathbb{R}^n/\mathcal{K}(P)$ and define the induced linear map $\bar{P}:Q\to Q$ by $\bar{P}([v]) \coloneqq [Pv]$. Then we have $\rho(\bar{P}) < 1$. Moreover, for any $\gamma\in(\rho(\bar{P}),1)$ there exists a norm $\|\cdot\|_{\mathrm q}$ on $Q$ such that We also use $\|\cdot\|_{\mathrm q}$ to denote the pullback semi-norm on $\mathbb{R}^n$ defined by $\|v\|_{\mathrm q}\coloneqq

Figures (1)

  • Figure 1: Avg-reward profile error ($\ell_\infty$) versus TD iteration $t$ for six fixed general-chain MRPs (Table \ref{['tab:mrp_suite']}). Each panel corresponds to one $P\in\mathbb{R}^{n\times n}$; curves are mean $\pm$ one standard deviation over sampling seeds.

Theorems & Definitions (56)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Definition 1
  • Lemma 1
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • Theorem 4
  • Corollary 1: End to end query complexity
  • ...and 46 more