Span-Based Optimal Sample Complexity for Weakly Communicating and General Average Reward MDPs

TL;DR

This work resolves the sample complexity of learning $ extvarepsilon$-optimal policies for average-reward MDPs under a generative model, first achieving minimax-optimal rates for weakly communicating AMDPs with $ ilde{O}(SAH/ ext{ε}^2)$ samples. It introduces a general AMDP framework requiring a new transient-time parameter $B$, obtaining $ ilde{O}(SA(B+H)/ ext{ε}^2)$ and matching minimax lower bounds, by reducing AMDPs to discounted MDPs and developing refined variance bounds for discounted cases. The analysis sharpens the horizon dependence from cubic to near-quadratic in fixed instances and provides improved discounted-MDP bounds $ ilde{O}(SAH/(1- ext{γ})^2 ext{ε}^2)$ and $ ilde{O}(SA(B+H)/(1- ext{γ})^2 ext{ε}^2)$. A key limitation is the necessity of knowledge of $H$ (and in general AMDPs, the bound $B$), with open questions about removing these prerequisites and extending to online settings. Overall, the paper deepens the connection between average-reward and discounted formulations and establishes tight instance-dependent sample complexities for both weakly communicating and general MDPs.

Abstract

We study the sample complexity of learning an $\varepsilon$-optimal policy in an average-reward Markov decision process (MDP) under a generative model. For weakly communicating MDPs, we establish the complexity bound $\widetilde{O}(SA\frac{H}{\varepsilon^2} )$, where $H$ is the span of the bias function of the optimal policy and $SA$ is the cardinality of the state-action space. Our result is the first that is minimax optimal (up to log factors) in all parameters $S,A,H$, and $\varepsilon$, improving on existing work that either assumes uniformly bounded mixing times for all policies or has suboptimal dependence on the parameters. We also initiate the study of sample complexity in general (multichain) average-reward MDPs. We argue a new transient time parameter $B$ is necessary, establish an $\widetilde{O}(SA\frac{B + H}{\varepsilon^2})$ complexity bound, and prove a matching (up to log factors) minimax lower bound. Both results are based on reducing the average-reward MDP to a discounted MDP, which requires new ideas in the general setting. To optimally analyze this reduction, we develop improved bounds for $γ$-discounted MDPs, showing that $\widetilde{O}(SA\frac{H}{(1-γ)^2\varepsilon^2} )$ and $\widetilde{O}(SA\frac{B + H}{(1-γ)^2\varepsilon^2} )$ samples suffice to learn $\varepsilon$-optimal policies in weakly communicating and in general MDPs, respectively. Both these results circumvent the well-known minimax lower bound of $\widetildeΩ(SA\frac{1}{(1-γ)^3\varepsilon^2} )$ for $γ$-discounted MDPs, and establish a quadratic rather than cubic horizon dependence for a fixed MDP instance.

Figures (3)

• Figure 1: A general MDP where $\gamma$-discounted approximation fails unless $\frac{1}{1-\gamma} = \Omega(T) \gg \left\Vert h^\star\right\Vert _{\textnormal{span}}$.
• Figure 2: MDPs used in Theorem \ref{['thm:impossibility_H_estimation']}
• Figure 3: MDP Instances Used in the Proof of Lower Bound in Theorem \ref{['thm:lower_bound_general2']}

Theorems & Definitions (55)

• Theorem 1: Sample Complexity of Weakly Communicating DMDP
• proof : Proof highlights for Theorem \ref{['thm:DMDP_bound']}
• Theorem 2: Sample Complexity of Weakly Communicating AMDP
• Theorem 3
• Theorem 4: Lower Bound for General AMDPs
• Theorem 5: Lower Bound for General DMDP
• Theorem 6: Average-to-Discount Reduction for General MDP
• proof : Proof highlights
• Theorem 7: Sample Complexity of General DMDP
• Theorem 8: Sample Complexity of General AMDP