Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Computing Fixpoints of Learned Functions: Chaotic Iteration and Simple Stochastic Games

Paolo Baldan, Sebastian Gurke, Barbara König, Florian Wittbold

TL;DR

The paper tackles the problem of computing the least fixpoint $oldsymbol{}$ of monotone, non-expansive functions on $oldsymbol{X} ooldsymbol{X}$ in high dimensions when the target function is only available through approximations $(f_n)$. It generalizes the dampened Mann iteration by allowing learning rates $oldsymbol{ ome_n}$ to converge to $0$ or fail to converge and by enabling chaotic, dimension-aware updates, thereby improving scalability and adaptability to varying convergence rates. The authors prove convergence of these generalized schemes for simple stochastic games (SSGs) and their sampled approximations, and connect the theory to Markov decision processes (MDPs) and behavioral metrics, including multi-step and subfunction extensions. The results provide a robust framework for fixed-point computation in quantitative models, with practical implications for reinforcement learning, robust distance measures, and scalable planning in high-dimensional settings, while also suggesting avenues for data-driven chaotic update strategies and parallel implementations.

Abstract

The problem of determining the (least) fixpoint of (higher-dimensional) functions over the non-negative reals frequently occurs when dealing with systems endowed with a quantitative semantics. We focus on the situation in which the functions of interest are not known precisely but can only be approximated. As a first contribution we generalize an iteration scheme called dampened Mann iteration, recently introduced in the literature. The improved scheme relaxes previous constraints on parameter sequences, allowing learning rates to converge to zero or not converge at all. While seemingly minor, this flexibility is essential to enable the implementation of chaotic iterations, where only a subset of components is updated in each step, allowing to tackle higher-dimensional problems. Additionally, by allowing learning rates to converge to zero, we can relax conditions on the convergence speed of function approximations, making the method more adaptable to various scenarios. We also show that dampened Mann iteration applies immediately to compute the expected payoff in various probabilistic models, including simple stochastic games, not covered by previous work.

Computing Fixpoints of Learned Functions: Chaotic Iteration and Simple Stochastic Games

TL;DR

The paper tackles the problem of computing the least fixpoint of monotone, non-expansive functions on in high dimensions when the target function is only available through approximations . It generalizes the dampened Mann iteration by allowing learning rates to converge to or fail to converge and by enabling chaotic, dimension-aware updates, thereby improving scalability and adaptability to varying convergence rates. The authors prove convergence of these generalized schemes for simple stochastic games (SSGs) and their sampled approximations, and connect the theory to Markov decision processes (MDPs) and behavioral metrics, including multi-step and subfunction extensions. The results provide a robust framework for fixed-point computation in quantitative models, with practical implications for reinforcement learning, robust distance measures, and scalable planning in high-dimensional settings, while also suggesting avenues for data-driven chaotic update strategies and parallel implementations.

Abstract

The problem of determining the (least) fixpoint of (higher-dimensional) functions over the non-negative reals frequently occurs when dealing with systems endowed with a quantitative semantics. We focus on the situation in which the functions of interest are not known precisely but can only be approximated. As a first contribution we generalize an iteration scheme called dampened Mann iteration, recently introduced in the literature. The improved scheme relaxes previous constraints on parameter sequences, allowing learning rates to converge to zero or not converge at all. While seemingly minor, this flexibility is essential to enable the implementation of chaotic iterations, where only a subset of components is updated in each step, allowing to tackle higher-dimensional problems. Additionally, by allowing learning rates to converge to zero, we can relax conditions on the convergence speed of function approximations, making the method more adaptable to various scenarios. We also show that dampened Mann iteration applies immediately to compute the expected payoff in various probabilistic models, including simple stochastic games, not covered by previous work.
Paper Structure (15 sections, 4 theorems, 86 equations, 2 figures, 1 table)

This paper contains 15 sections, 4 theorems, 86 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Preliminaries and Notation
  3. Dampened Mann iteration.
  4. Generalizing Dampened Mann Iteration
  5. Schemes With Non-Converging Learning Rates
  6. Chaotic Dampened-Mann Iteration
  7. Chaotic random iteration.
  8. Working with Approximations
  9. Applications to State-based Systems
  10. Simple Stochastic Games
  11. Convergence for Simple Stochastic Games
  12. Numerical Experiments
  13. Vanishing and diverging learning rates.
  14. Chaotic iteration.
  15. Conclusion

Key Result

corollary 1

Let $f\colon X \to X$ be a monotone and non-expansive map and $(f_n)$ be a sequence of maps $f_n\colon X \to X$ with $\|f_n - f\| \to 0$. Then, there are sequences $(\alpha_n)$ and $(\beta_n)$ such that, for every initial value $x_0\in X$, the sequence defined by eq:iteration converges to $\mu f$.

Figures (2)

  • Figure 1: Results for the experiments of non-chaotic iterations on 50 randomly generated SSGs. Lines denote the mean error, the half-opaque area the 25th to 75th percentile, the weakly visible area the minimum to maximum area. Results for $\mathcal{S}_1$ and $\mathcal{S}_4$ are only barely visible as they are almost identical to the ones from $\mathcal{S}_2$ and $\mathcal{S}_6$, respectively.
  • Figure 2: Results for the experiments of chaotic iterations on 50 randomly generated SSGs. Again, the results for $\mathcal{S}_1$ and $\mathcal{S}_4$ are only barely visible as they are almost identical to the ones from $\mathcal{S}_2$ and $\mathcal{S}_6$, respectively.

Theorems & Definitions (35)

  • definition 1: Mann scheme
  • definition 2: Progressing scheme
  • proof
  • proof
  • corollary 1
  • proof
  • remark 1
  • definition 3: Generalized Mann scheme
  • definition 4: Progressing generalized Mann scheme
  • proof
  • ...and 25 more