Bidding Games on Markov Decision Processes with Quantitative Reachability Objectives

TL;DR

This work introduces bidding games on finite Markov decision processes with reachability objectives, where two players with budgets summing to 1 bid at control vertices to influence the next action and aim to maximize or minimize the probability of reaching a target. It develops a two-dimensional bounded-horizon value iteration that yields monotone staircases representing reachable budget–probability pairs, and proves that the unbounded-horizon threshold Th$_v$ is the completely separating intersection of limiting sets. The authors provide exact solutions for acyclic MDPs, show connections to simple stochastic games for lower bounds, and present approximations via abstraction (AlgApprox) and exact semi-decision procedures (AlgExact) under certain assumptions. The framework supports applications in auction-based scheduling and modular multi-objective decision-making under uncertainty, while leaving key decidability and complexity questions open for the general case. Overall, the paper advances a principled method to analyze and synthesize strategies in stochastic, budget-constrained bidding environments with practical relevance to decentralized resource allocation and scheduling.

Abstract

Graph games are fundamental in strategic reasoning of multi-agent systems and their environments. We study a new family of graph games which combine stochastic environmental uncertainties and auction-based interactions among the agents, formalized as bidding games on (finite) Markov decision processes (MDP). Normally, on MDPs, a single decision-maker chooses a sequence of actions, producing a probability distribution over infinite paths. In bidding games on MDPs, two players -- called the reachability and safety players -- bid for the privilege of choosing the next action at each step. The reachability player's goal is to maximize the probability of reaching a target vertex, whereas the safety player's goal is to minimize it. These games generalize traditional bidding games on graphs, and the existing analysis techniques do not extend. For instance, the central property of traditional bidding games is the existence of a threshold budget, which is a necessary and sufficient budget to guarantee winning for the reachability player. For MDPs, the threshold becomes a relation between the budgets and probabilities of reaching the target. We devise value-iteration algorithms that approximate thresholds and optimal policies for general MDPs, and compute the exact solutions for acyclic MDPs, and show that finding thresholds is at least as hard as solving simple-stochastic games.

Figures (2)

• Figure 1: Value iteration for the game on the left with the sequence of reachability values ($c$ is the target) on the right.
• Figure 2: Illustration of the $\mathcal{T}_v$ operator for when $v\in {V_\mathsf{r}}$ (left) and $v\in{V_\mathsf{c}}$ (right). In both cases, we assume there are two successors whose values from the $(i-1)$-th iteration are given as the red and blue regions. The outputs of $\mathcal{T}_v$ is shown as the set with thick boundaries. For $v\in{V_\mathsf{r}}$ (left), we assume uniform transition probabilities (i.e., $0.5$ for each successor).

Theorems & Definitions (19)

• definition 1: Threshold
• definition 2: Completely separating thresholds
• theorem 1
• lemma 1
• corollary 1
• corollary 2
• theorem 2
• corollary 3: Determinacy
• lemma 2
• theorem 3