Epsilon-Optimal Policies for Average-Cost Separable MDPs with Perturbations

TL;DR

This paper tackles infinite-horizon average-cost MDPs with rewards and transitions that are nearly separable. It first derives an explicit, state-independent optimal policy for the totally separable baseline ($\varepsilon=0$), with $a^* = \arg\max_a [\, \pi_a^\top r_S + r_A(a) \,]$ and $g^* = \max_a [\, \pi_a^\top r_S + r_A(a) \,]$, and shows the average-cost optimality equation holds exactly. It then proves a first-order perturbation result: when the separable structure is perturbed by $\varepsilon>0$, the same constant policy remains $\mathcal{O}(\varepsilon)$-optimal, with bounds derived from invariant-distribution perturbations and linearized bias equations. The contributions are an explicit closed-form optimal policy in the baseline and a rigorous robustness bound under perturbations, providing quantitative guarantees for near-separable MDPs. This establishes a practical robustness result for long-run decision-making in structured systems and informs extensions to discounted, constrained, and learning settings with model uncertainty.

Abstract

We study a class of infinite-horizon average-cost Markov Decision Processes (MDPs) whose reward and transition structures are nearly separable. For the totally separable baseline (that is, with no perturbation), we derive an explicit stationary decision rule that is exactly average-optimal. We then show that under an epsilon-perturbation of the separable structure, this policy remains epsilon-optimal, meaning that the loss in the average reward is of order O(epsilon).