Fitted Q Evaluation Without Bellman Completeness via Stationary Weighting

TL;DR

The paper addresses instability in off-policy value evaluation caused by a norm mismatch between the regression objective and the Bellman contraction geometry. It proposes stationary-weighted FQE, which reweights each regression step by an estimate of the stationary density ratio to align with the $L^2(\mu)$ geometry where the Bellman operator is a $\gamma$-contraction. The authors establish contraction-based guarantees without assuming realizability or Bellman completeness, provide finite-sample error bounds that decompose statistical and density-ratio effects, and show robustness to misspecification, including reward misspecification. They demonstrate empirically that stationary weighting restores geometric decay and prevents exponential divergence in challenging norm-mismatch scenarios, while adding minimal computational overhead. The approach offers a practical, theoretically grounded refinement to regression-based off-policy evaluation with potential applications in offline RL and beyond, and a companion work extends the idea to soft FQI in the control setting.

Abstract

Fitted Q-evaluation (FQE) is a central method for off-policy evaluation in reinforcement learning, but it generally requires Bellman completeness: that the hypothesis class is closed under the evaluation Bellman operator. This requirement is challenging because enlarging the hypothesis class can worsen completeness. We show that the need for this assumption stems from a fundamental norm mismatch: the Bellman operator is gamma-contractive under the stationary distribution of the target policy, whereas FQE minimizes Bellman error under the behavior distribution. We propose a simple fix: reweight each regression step using an estimate of the stationary density ratio, thereby aligning FQE with the norm in which the Bellman operator contracts. This enables strong evaluation guarantees in the absence of realizability or Bellman completeness, avoiding the geometric error blow-up of standard FQE in this setting while maintaining the practicality of regression-based evaluation.

Figures (8)

• Figure 1: Norm mismatch: $Q \in \mathcal{F}$ (red), $\mathcal{T}Q$ leaves $\mathcal{F}$ but lies in $L^2(\mu)$ (blue), and $\Pi_{\mathcal{F}}^{\nu_b}$ projects it back under the behavior-norm $L^2(\nu_b)$. The resulting composite map $\mathcal{U}_{\nu_b} = \Pi_{\mathcal{F}}^{\nu_b}\mathcal{T}$ need not be contractive.
• Figure 2: Moderate-overlap regime ($\gamma=0.95$): final error vs. $\kappa$ (left) and iteration curves (right).
• Figure 3: Severe norm-mismatch regime: final error vs. $\gamma$ (left) and iteration curves (right).
• Figure 4: Final stationary-norm error versus stationary overlap for $\gamma=0.95$. Left: $\|Q^{(K)} - Q^\star\|_{2,\mu}$ across $\kappa$. Right: iteration curves at selected overlap levels.
• Figure 5: High-discount, low-overlap regime ($\gamma=0.999$). Left: final error vs. $\kappa$. Right: iteration curves at $\kappa=0.05$.

Theorems & Definitions (26)

• Lemma 1: Contraction of the Bellman operator
• Theorem 1: Function approximation error
• Lemma 2: Approximate Picard Iteration Error Bound
• Lemma 3: Regret for inexact Picard iteration
• Theorem 2: Estimation Error for Stationary-Weighted FQE
• Theorem 3: Reward Misspecification
• Lemma 4: Bellman contraction for $Q$-functions under a stationary measure
• proof
• Lemma 5: Contraction of the projected Bellman operator
• proof