A Unifying View of Coverage in Linear Off-Policy Evaluation

TL;DR

The paper addresses linear off-policy evaluation under minimal realizability and identifies fragmented coverage notions as a core obstacle. It introduces feature-dynamics coverage $C^{\pi}_{\phi}$ along with an IV-inspired interpretation of LSTDQ, providing finite-sample bounds that separate population and empirical regimes via $C^{\pi}_{\phi}$ and $\widehat{C}^{\pi}_{\phi}$. The results unify several existing notions, showing that $C^{\pi}_{\phi}$ recovers standard linear coverage under Bellman completeness and aggregates to aggregated concentrability in abstractions, while linking to marginalized importance sampling and BRM. This framework clarifies when linear OPE is tractable and offers directionally sharp, dimension-free guarantees in key regimes, with broad implications for offline RL and model-selection tasks.

Abstract

Off-policy evaluation (OPE) is a fundamental task in reinforcement learning (RL). In the classic setting of linear OPE, finite-sample guarantees often take the form $$ \textrm{Evaluation error} \le \textrm{poly}(C^π, d, 1/n,\log(1/δ)), $$ where $d$ is the dimension of the features and $C^π$ is a coverage parameter that characterizes the degree to which the visited features lie in the span of the data distribution. While such guarantees are well-understood for several popular algorithms under stronger assumptions (e.g. Bellman completeness), the understanding is lacking and fragmented in the minimal setting where only the target value function is linearly realizable in the features. Despite recent interest in tight characterizations of the statistical rate in this setting, the right notion of coverage remains unclear, and candidate definitions from prior analyses have undesirable properties and are starkly disconnected from more standard definitions in the literature. We provide a novel finite-sample analysis of a canonical algorithm for this setting, LSTDQ. Inspired by an instrumental-variable view, we develop error bounds that depend on a novel coverage parameter, the feature-dynamics coverage, which can be interpreted as linear coverage in an induced dynamical system for feature evolution. With further assumptions -- such as Bellman-completeness -- our definition successfully recovers the coverage parameters specialized to those settings, finally yielding a unified understanding for coverage in linear OPE.

Figures (1)

• Figure 1: Illustration of the evolution of occupancies under the true dynamics $P^\pi$ (top row) and that of features under the compressed dynamics $B^\pi$ (bottom row). Under Bellman completeness, the dashed blue arrows hold and two routes ($\to \ldots \to {\color{blue} \downarrow}$ vs. ${\color{blue} \downarrow} {\color{red}\to \ldots \to}$) yield the same expected feature vectors, but they are generally different without such an assumption.

Theorems & Definitions (42)

• theorem 1: Population Coverage Bound
• theorem 2: Empirical Coverage Bound
• proposition 1
• proposition 2
• proposition 3
• definition 1: Aggregated concentrability
• proposition 4
• proposition 5
• theorem 2: Empirical Coverage Bound
• proof : thm:mainthm-emp