Fast Non-Episodic Finite-Horizon RL with K-Step Lookahead Thresholding

TL;DR

The paper addresses non-episodic, finite-horizon RL by shifting from full-horizon value estimation to a K-step lookahead Q-function, combined with a thresholding rule to prune suboptimal actions. The LGKT framework provides online learning algorithms with provable regret guarantees: minimax optimal constant regret for K=1 and sublinear regret bounds of order $\max\{(K-1)\sqrt{SAT}, C_{K-1}\sqrt{SAT\log(T)}\}$ for general $K$, enabling fast convergence in practice. The authors validate the approach across synthetic MDPs, JumpRiverSwim, FrozenLake, and AnyTrading, demonstrating superior cumulative rewards and robust performance. This work offers a practical, theory-backed method for efficient planning in non-episodic finite-horizon settings and suggests promising directions for extending to more complex, continuous domains. The key innovation is balancing lookahead depth with estimation variance through adaptive thresholds and controlled exploration, yielding strong finite-sample guarantees and empirical gains.

Abstract

Online reinforcement learning in non-episodic, finite-horizon MDPs remains underexplored and is challenged by the need to estimate returns to a fixed terminal time. Existing infinite-horizon methods, which often rely on discounted contraction, do not naturally account for this fixed-horizon structure. We introduce a modified Q-function: rather than targeting the full-horizon, we learn a K-step lookahead Q-function that truncates planning to the next K steps. To further improve sample efficiency, we introduce a thresholding mechanism: actions are selected only when their estimated K-step lookahead value exceeds a time-varying threshold. We provide an efficient tabular learning algorithm for this novel objective, proving it achieves fast finite-sample convergence: it achieves minimax optimal constant regret for $K=1$ and $\mathcal{O}(\max((K-1),C_{K-1})\sqrt{SAT\log(T)})$ regret for any $K \geq 2$. We numerically evaluate the performance of our algorithm under the objective of maximizing reward. Our implementation adaptively increases K over time, balancing lookahead depth against estimation variance. Empirical results demonstrate superior cumulative rewards over state-of-the-art tabular RL methods across synthetic MDPs and RL environments: JumpRiverswim, FrozenLake and AnyTrading.

Figures (10)

• Figure 1: Running average reward over 1000 distinct MDP instances. Left: $S=10,A=5$, Right: $S=100,A=25$. Right excludes PMEVI-KLUCRL due to its prohibitive runtime and on Right, KLUCRL overlaps with UCRL2.
• Figure 2: Running average reward under JumpRiverSwim environments averaged over 100 repetitions. Top Left: 5 states, Top Right: 8 states, Bottom Left: 15 states.
• Figure 3: Left: Running average reward under FrozenLake environments with $4\times 4$ state space averaged over 100 repetitions. LG1-2T has parameter $t_c=10000$, Right: Cumulative reward of 20000 environment steps under AnyTrading environment averaged over 1000 repetitions. LG1-2T has parameter $t_c=30$. Right exclude model based methods for prohibitive long runtime.
• Figure 4: The FrozenLake environment.
• Figure 5: Ablations on 1000 synthetic MDPs. $S=100,A=25$

Theorems & Definitions (13)

• Remark 2.2: Hardness of Applying Infinite-horizon Algorithms
• Definition 3.1: K-Step Lookahead Reward
• Theorem 3.3
• Theorem 3.4
• Theorem 4.2
• Theorem 4.4
• Corollary 4.5
• Remark 4.6: Impact of K On Convergence Rate
• Lemma 4.1
• proof