Stationary Reweighting Yields Local Convergence of Soft Fitted Q-Iteration

TL;DR

This paper identifies a fundamental source of instability in offline, function-approximation RL: the regression updates operate under the behavior distribution, while the soft Bellman operator contracts in the stationary distribution of the soft-optimal policy. It proposes stationary-reweighted soft FQI, which uses stationary-density-based weights to align the regression geometry with the operator's natural norm, enabling local contraction and linear convergence under approximate realizability. The authors also develop a finite-sample theory that accounts for sampling error, weight estimation, and misspecification, and they propose a temperature-homotopy strategy to move toward global convergence, potentially reaching the hard-max regime under a margin condition. Empirically, stationary reweighting stabilizes soft FQI under severe norm-mismatch and benefits from temperature continuation, suggesting practical pathways for robust offline value iteration without Bellman completeness.

Abstract

Fitted Q-iteration (FQI) and its entropy-regularized variant, soft FQI, are central tools for value-based model-free offline reinforcement learning, but can behave poorly under function approximation and distribution shift. In the entropy-regularized setting, we show that the soft Bellman operator is locally contractive in the stationary norm of the soft-optimal policy, rather than in the behavior norm used by standard FQI. This geometric mismatch explains the instability of soft Q-iteration with function approximation in the absence of Bellman completeness. To restore contraction, we introduce stationary-reweighted soft FQI, which reweights each regression update using the stationary distribution of the current policy. We prove local linear convergence under function approximation with geometrically damped weight-estimation errors, assuming approximate realizability. Our analysis further suggests that global convergence may be recovered by gradually reducing the softmax temperature, and that this continuation approach can extend to the hardmax limit under a mild margin condition.

Figures (2)

• Figure 1: Contraction geometry. Left: contraction region $\mathbb{B}(r_0, Q^\star)$ and basin of attraction $\mathbb{B}(r_{\max}, Q^\star_{\mathcal{F}})$. Right: one–dimensional curvature slice of $\mathcal{T}(Q)-\mathcal{T}_{\pi_\star}^{\mathrm{eval}}(Q)$ near $Q^\star$, with distances $r^\star = d(Q, Q^\star)$ and $r = d(Q, Q^\star_{\mathcal{F}})$.
• Figure 2: Soft fitted $Q$-iteration under severe norm mismatch on a Garnet MDP. Left: direct training at $\tau$. Middle and right: temperature homotopy. Shaded regions denote the 25%--75% quantiles across random seeds, and the dashed vertical line marks when $\tau$ reaches its target value.

Theorems & Definitions (38)

• Lemma 1: First derivative of the soft Bellman operator
• Lemma 2: Second-order remainder bound
• Theorem 1: Local contraction of the soft Bellman operator
• Lemma 3: Existence and local uniqueness of $Q^\star_{\mathcal{F}}$
• Theorem 2: Local linear convergence around the projected soft optimum
• Corollary 1: Finite-time tightening of the contraction rate
• Lemma 4: Inexact iteration error bound
• Lemma 5: Statistical accuracy of one weighted regression
• Theorem 3: Local convergence with geometrically damped weight error
• Theorem 4: Local convergence with decoupled errors