$K$-Nearest-Neighbor Resampling for Off-Policy Evaluation in Stochastic Control

TL;DR

This work introduces a K-nearest-neighbor resampling approach for off-policy evaluation in stochastic control with episodic data, enabling trajectory-based counterfactual estimation without modeling the environment. By extending Stone's Theorem to episodic, counterfactual settings and establishing consistency for the KNN path and KNNR estimators, the method guarantees vanishing MSE as data grows, under weak conditions. The approach is model-free, parallelizable, and efficient via tree-based nearest-neighbor search, with competitive performance against baselines in LQR, limit-order-book trading, and stochastic bin packing. The results suggest strong practical impact for evaluating deterministic-feedback policies in continuous-state, stochastic environments where traditional iid-transition assumptions fail.

Abstract

In this paper, we propose a novel $K$-nearest neighbor resampling procedure for estimating the performance of a policy from historical data containing realized episodes of a decision process generated under a different policy. We provide statistical consistency results under weak conditions. In particular, we avoid the common assumption of identically and independently distributed transitions and rewards. Instead, our analysis allows for the sampling of entire episodes, as is common practice in most applications. To establish the consistency in this setting, we generalize Stone's Theorem, a well-known result in nonparametric statistics on local averaging, to include episodic data and the counterfactual estimation underlying off-policy evaluation (OPE). By focusing on feedback policies that depend deterministically on the current state in environments with continuous state-action spaces and system-inherent stochasticity effected by chosen actions, and relying on trajectory simulation similar to Monte Carlo methods, the proposed method is particularly well suited for stochastic control environments. Compared to other OPE methods, our algorithm does not require optimization, can be efficiently implemented via tree-based nearest neighbor search and parallelization, and does not explicitly assume a parametric model for the environment's dynamics. Numerical experiments demonstrate the effectiveness of the algorithm compared to existing baselines in a variety of stochastic control settings, including a linear quadratic regulator, trade execution in limit order books, and online stochastic bin packing.

Figures (2)

• Figure 1: Visualization of the $K$-NN path with $k^3=(3,2,4)$ where $S$ is projected onto a line for each $t$ and metric distances are assumed to be preserved in this one dimensional representation. The circles represent samples and the filled circle denotes the sample that is part of the $K$-NN path. The crosses represent the starting point on the first line and the transitions from samples on the $K$-NN path on the subsequent lines.
• Figure 2: Experimental results: Each column corresponds to one test environment. First row: Median MSEs of the value estimates (where the target for the value is computed via Monte Carlo roll outs using the target policy) versus the number of episodes sampled under the behaviour policy (log-log plot). Second row: Standard deviation in the resampling estimates of the value as a function of the number of episodes in the data set. Third row: Comparison of median runtimes for the resampling algorithm and MFMC depending on the number of episodes in the data set.

Theorems & Definitions (25)

• Remark 1
• Remark 2
• Remark 3
• Theorem 1
• Theorem 2
• Remark 4
• Theorem 3
• proof : Proof of Theorem \ref{['thm:stone_mult']}
• Proposition 1
• proof