Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Piecewise Affine Reachability with Bellman Operators

Anton Varonka, Kazuki Watanabe

TL;DR

This work studies reachability for piecewise affine maps that arise as Bellman operators of Markov decision processes, focusing on the BOR problem: given $\mathbf{s},\mathbf{t}\in[0,1]^d\cap\mathbb{Q}^d$ and a Bellman operator $\Phi$ with a unique fixed point $\boldsymbol{\mu}\Phi$, can we reach $\mathbf{t}$ by iterated application of $\Phi$ starting from $\mathbf{s}$? The authors exploit the contraction property and the fixed-point structure to obtain decidability results across dimensions: BOR is decidable for any $d$ when $\mathbf{t}\neq\boldsymbol{\mu}\Phi$, and decidable when $\mathbf{t}=\boldsymbol{\mu}\Phi$ and $\mathbf{s}$ is comparable to $\boldsymbol{\mu}\Phi$; moreover, BOR is decidable in dimension $2$ for arbitrary $\mathbf{s},\mathbf{t}$ even when incomparable. The core techniques include sign abstractions for below/above fixed-point regions, neighborhood-tight action analysis, and a reduction to matrix-semigroup reachability (PFR) in the incomparable case, with a complete decidability argument in 2D leveraging a total order on action-induced lines. These results significantly expand the known decidability frontier for PAM reachability by identifying a broad and natural PAM subclass (Bellman operators) with favorable algorithmic properties. The work also discusses mortality and connections to general PAMs, outlining directions for future work beyond Bellman operators.

Abstract

A piecewise affine map is one of the simplest mathematical objects exhibiting complex dynamics. The reachability problem of piecewise affine maps is as follows: Given two vectors $\mathbf{s}, \mathbf{t} \in \mathbb{Q}^d$ and a piecewise affine map $f$, is there $n\in \mathbb{N}$ such that $f^{n}(\mathbf{s}) = \mathbf{t}$? Koiran, Cosnard, and Garzon show that the reachability problem of piecewise affine maps is undecidable even in dimension 2. Most of the recent progress has been focused on decision procedures for one-dimensional piecewise affine maps, where the reachability problem has been shown to be decidable for some subclasses. However, the general undecidability discouraged research into positive results in arbitrary dimension. In this work, we investigate a rich subclass of piecewise affine maps arising as Bellman operators of Markov decision processes (MDPs). We consider the reachability problem restricted to this subclass and examine its decidability in arbitrary dimensions. We establish that the reachability problem for Bellman operators is decidable in any dimension under either of the following conditions (i) the target vector $\mathbf{t}$ is not the fixed point of the operator $f$; or (ii) the initial and target vectors $\mathbf{s}$ and $\mathbf{t}$ are comparable with respect to the componentwise order. Furthermore, we show that the reachability problem for two-dimensional Bellman operators is decidable for arbitrary $\mathbf{s}, \mathbf{t}\in \mathbb{Q}^d$, in contrast to the known undecidability of reachability for general piecewise affine maps.

On Piecewise Affine Reachability with Bellman Operators

TL;DR

This work studies reachability for piecewise affine maps that arise as Bellman operators of Markov decision processes, focusing on the BOR problem: given and a Bellman operator with a unique fixed point , can we reach by iterated application of starting from ? The authors exploit the contraction property and the fixed-point structure to obtain decidability results across dimensions: BOR is decidable for any when , and decidable when and is comparable to ; moreover, BOR is decidable in dimension for arbitrary even when incomparable. The core techniques include sign abstractions for below/above fixed-point regions, neighborhood-tight action analysis, and a reduction to matrix-semigroup reachability (PFR) in the incomparable case, with a complete decidability argument in 2D leveraging a total order on action-induced lines. These results significantly expand the known decidability frontier for PAM reachability by identifying a broad and natural PAM subclass (Bellman operators) with favorable algorithmic properties. The work also discusses mortality and connections to general PAMs, outlining directions for future work beyond Bellman operators.

Abstract

A piecewise affine map is one of the simplest mathematical objects exhibiting complex dynamics. The reachability problem of piecewise affine maps is as follows: Given two vectors and a piecewise affine map , is there such that ? Koiran, Cosnard, and Garzon show that the reachability problem of piecewise affine maps is undecidable even in dimension 2. Most of the recent progress has been focused on decision procedures for one-dimensional piecewise affine maps, where the reachability problem has been shown to be decidable for some subclasses. However, the general undecidability discouraged research into positive results in arbitrary dimension. In this work, we investigate a rich subclass of piecewise affine maps arising as Bellman operators of Markov decision processes (MDPs). We consider the reachability problem restricted to this subclass and examine its decidability in arbitrary dimensions. We establish that the reachability problem for Bellman operators is decidable in any dimension under either of the following conditions (i) the target vector is not the fixed point of the operator ; or (ii) the initial and target vectors and are comparable with respect to the componentwise order. Furthermore, we show that the reachability problem for two-dimensional Bellman operators is decidable for arbitrary , in contrast to the known undecidability of reachability for general piecewise affine maps.

Paper Structure

This paper contains 15 sections, 18 theorems, 12 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Our Approach.
  3. Contributions.
  4. Background
  5. Preliminary
  6. Target problem
  7. Bellman Operator Reachability in Arbitrary Dimension
  8. Target vector that is not the fixed point
  9. Initial vector below the fixed point
  10. Initial vector above the fixed point
  11. Initial and target vectors are incomparable
  12. A Decidable Case $d=2$
  13. Actions in $d=2$ and lines.
  14. Related Work and Discussion
  15. Omitted Proof

Key Result

Proposition 5

There exists an optimal positional scheduler $\sigma_\text{pos} \in \Sigma$ such that $\mathbb{P}^{\sigma_\text{pos}}(s \models \lozenge t) = p_s$ holds for all $s \in S$.

Figures (4)

  • Figure 1: An MDP $\mathcal{M}_1$.
  • Figure 2: An MDP $\mathcal{M}_2$.
  • Figure 3: An MDP $\mathcal{M}_3$ of \ref{['ex:2d']}.
  • Figure 4: $\mathcal{M}_3$ illustrating case 4.b.i.

Theorems & Definitions (34)

  • Example 1
  • Definition 2: MDP Puterman94
  • Example 3
  • Definition 4: reachability probability
  • Proposition 5: e.g. bk
  • Definition 6: Bellman operator
  • Example 7
  • Proposition 8: bkchatterjee2008viPuterman94
  • Proposition 9: HaddadM18
  • Definition 10: end component Alfaro97
  • ...and 24 more