The BoBW Algorithms for Heavy-Tailed MDPs

TL;DR

This work develops the first Best-of-Both-Worlds (BoBW) framework for episodic Markov decision processes with heavy-tailed feedback (HTMDPs), where losses have bounded α-moments (α∈(1,2]). It introduces HT-FTRL-OM for known transitions and HT-FTRL-UOB for unknown transitions, both built on Follow-The-Regularized-Leader over occupancy measures with Tsallis entropy and robust skipping estimators. The authors prove BoBW regret bounds in adversarial and self-bounding (including stochastic) regimes, with instance-independent rates in the adversarial setting and logarithmic, problem-dependent rates in stochastic regimes; for the unknown-transition case they provide the first BoBW results with $ ilde{O}(T^{1/α}+ ilde{O}( oot ext{log}{T}))$-type scalings and $ ilde{O}( ext{log}^2 T)$ stochastic bounds. Central innovations include a local heavy-tailed loss shifting analysis, a suboptimal-mass propagation principle, and a regret decomposition that isolates transition uncertainty from heavy-tailed estimation and skipping bias. Collectively, these results advance robust, regime-adaptive RL in HTMDPs and open avenues for tighter dependence on S, A, H and extensions to function approximation.

Abstract

We investigate episodic Markov Decision Processes with heavy-tailed feedback (HTMDPs). Existing approaches for HTMDPs are conservative in stochastic environments and lack adaptivity in adversarial regimes. In this work, we propose algorithms ```HT-FTRL-OM``` and ```HT-FTRL-UOB``` for HTMDPs that achieve Best-of-Both-Worlds (BoBW) guarantees: instance-independent regret in adversarial environments and logarithmic instance-dependent regret in self-bounding (including the stochastic case) environments. For the known transition setting, ```HT-FTRL-OM``` applies the Follow-The-Regularized-Leader (FTRL) framework over occupancy measures with novel skipping loss estimators, achieving a $\widetilde{\mathcal{O}}(T^{1/α})$ regret bound in adversarial regimes and a $\mathcal{O}(\log T)$ regret in stochastic regimes. Building upon this framework, we develop a novel algorithm ```HT-FTRL-UOB``` to tackle the more challenging unknown-transition setting. This algorithm employs a pessimistic skipping loss estimator and achieves a $\widetilde{\mathcal{O}}(T^{1/α} + \sqrt{T})$ regret in adversarial regimes and a $\mathcal{O}(\log^2(T))$ regret in stochastic regimes. Our analysis overcomes key barriers through several technical insights, including a local control mechanism for heavy-tailed shifted losses, a new suboptimal-mass propagation principle, and a novel regret decomposition that isolates transition uncertainty from heavy-tailed estimation errors and skipping bias.

Theorems & Definitions (48)

• Definition 3.1: Self-Bounding Constraint
• Theorem 4.1: BoBW Guarantees for the Known Transition Setting
• Lemma 4.2: Pointwise Uniform Bound of Shifted Loss
• Lemma 4.3: Second Moment Bound of Shifted Loss
• Lemma 4.4: Suboptimal Mass Propagation
• Theorem 5.1: BoBW Guarantee for the Unknown Transition Case
• Lemma 5.2
• Theorem B.1: Best-of-Both-Worlds Regret Guarantee
• Lemma B.2: The Uniform Bound on the Shifted Loss
• proof