Learning Markov Decision Processes under Fully Bandit Feedback

TL;DR

The paper tackles learning in episodic MDPs under fully bandit feedback, where only the episode total reward is observed. It introduces a phase-based successive-elimination algorithm that maintains per-state active action-sets and uses backward induction to refine choices, achieving regret on the order of $O\big(H^2 (A k)^H \sqrt{H k A T \log T}\big)$, with an explicit exponential dependence on the horizon $H$ that is shown to be necessary. For ordered MDPs, the authors obtain improved bounds $O\left(\left(\frac{2e A H^2}{k}\right)^k \sqrt{\frac{k^3 H}{A} T \log T}\right)$ by tailoring the exploration strategy to the structure. The work connects to classic stochastic optimization problems like the $k$-item prophet inequality and sequential posted pricing, providing near-optimal guarantees under restricted feedback, and empirical results demonstrate competitive performance against state-of-the-art semi-bandit methods despite far sparser feedback.

Abstract

A standard assumption in Reinforcement Learning is that the agent observes every visited state-action pair in the associated Markov Decision Process (MDP), along with the per-step rewards. Strong theoretical results are known in this setting, achieving nearly-tight $Θ(\sqrt{T})$-regret bounds. However, such detailed feedback can be unrealistic, and recent research has investigated more restricted settings such as trajectory feedback, where the agent observes all the visited state-action pairs, but only a single \emph{aggregate} reward. In this paper, we consider a far more restrictive ``fully bandit'' feedback model for episodic MDPs, where the agent does not even observe the visited state-action pairs -- it only learns the aggregate reward. We provide the first efficient bandit learning algorithm for episodic MDPs with $\widetilde{O}(\sqrt{T})$ regret. Our regret has an exponential dependence on the horizon length $\H$, which we show is necessary. We also obtain improved nearly-tight regret bounds for ``ordered'' MDPs; these can be used to model classical stochastic optimization problems such as $k$-item prophet inequality and sequential posted pricing. Finally, we evaluate the empirical performance of our algorithm for the setting of $k$-item prophet inequalities; despite the highly restricted feedback, our algorithm's performance is comparable to that of a state-of-art learning algorithm (UCB-VI) with detailed state-action feedback.

Figures (2)

• Figure 1: Ordered MDP instance with $k=2$ and $H =6$. Path $\bm{\tau}$ is from $(1,4)$ to $(6,1)$; it transitions downward at stages $2$ and $4$.
• Figure 2: Cumulative regret of algorithms on Prophet Inequality instances $\mathcal{I}_1$ (top two rows) and $\mathcal{I}_2$ (bottom two rows). Different values of $H$ are shown in the first and second rows for each type respectively.

Theorems & Definitions (27)

• Theorem 3.1
• Lemma 3.2
• proof
• Lemma 3.3
• proof
• Lemma 3.4
• Lemma 3.5
• proof
• Lemma 3.6
• proof