Is Q-learning an Ill-posed Problem?

TL;DR

The paper investigates instability in Q-learning within continuous-state environments. By progressively removing bootstrapping (via BSF-NFQ) and using true environment dynamics, it shows that instability persists even in simple benchmarks, and that the true Q-function exhibits abrupt discontinuities that NN function approximators struggle to capture. This reveals an ill-posed learning task where small changes in targets can yield large policy differences, suggesting that the problem is rooted in the MDP definition rather than the algorithm. The findings caution against assuming Q-learning is a universal remedy for reinforcement learning in continuous domains and highlight broader implications for offline policy evaluation and related methods that rely on sample-based Q-value evaluation.

Abstract

This paper investigates the instability of Q-learning in continuous environments, a challenge frequently encountered by practitioners. Traditionally, this instability is attributed to bootstrapping and regression model errors. Using a representative reinforcement learning benchmark, we systematically examine the effects of bootstrapping and model inaccuracies by incrementally eliminating these potential error sources. Our findings reveal that even in relatively simple benchmarks, the fundamental task of Q-learning - iteratively learning a Q-function from policy-specific target values - can be inherently ill-posed and prone to failure. These insights cast doubt on the reliability of Q-learning as a universal solution for reinforcement learning problems.

Figures (4)

• Figure 1: Iteration-wise policy performance averaged over 1,000 gym environment episodes. Blue lines represent the average return over 1,000 episodes each with 5,000 steps. Cross markers depict the quote of episodes reaching 5,000 steps. Green markers represent iterations where successful policies have been found.
• Figure 2: Iteration-wise policy performance averaged over 1,000 episodes. Average return of the original iterations are depicted in blue. The boxplots visualize policy performance results for retraining with saved Q-value targets on 100 seeds.
• Figure 3: Q-function values along 10,000 different pole angle values with cart position, cart velocity and pole velocity fixed at $0.0$. To calculate the rollouts, policies that are greedy with respect to their learned Q-value approximation were used, or for (c) $\epsilon$-greedy with an $\epsilon =0.05$. Note that the plots display only a single Q-value for each corresponding angle. Therefore, the significant differences between neighboring angle values indicate function discontinuities.
• Figure 4: Q-function values along 10,000 different pole angle values with cart position, cart velocity and pole velocity fixed at 0.0.