Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Replicable Reinforcement Learning with Linear Function Approximation

Eric Eaton, Marcel Hussing, Michael Kearns, Aaron Roth, Sikata Bela Sengupta, Jessica Sorrell

TL;DR

This paper tackles the problem of ensuring exact replicability in reinforcement learning when using linear function approximation in linear MDPs. It introduces two replicable tools—replicable ridge regression and replicable uncentered covariance estimation—and builds replicable RL algorithms for both generative-model and episodic settings (R-LSVI with core sets and R-LSVI-UCB) atop these tools. The authors provide finite-sample replicability guarantees and concrete sample complexities, supported by experiments on real datasets and neural Q-value quantization to illustrate practical stability benefits. Overall, the work advances reliable RL by enabling exact repeatability across runs, paving the way for safer and more auditable AI systems while outlining directions for future feature-learning and scalability to non-linear regimes.

Abstract

Replication of experimental results has been a challenge faced by many scientific disciplines, including the field of machine learning. Recent work on the theory of machine learning has formalized replicability as the demand that an algorithm produce identical outcomes when executed twice on different samples from the same distribution. Provably replicable algorithms are especially interesting for reinforcement learning (RL), where algorithms are known to be unstable in practice. While replicable algorithms exist for tabular RL settings, extending these guarantees to more practical function approximation settings has remained an open problem. In this work, we make progress by developing replicable methods for linear function approximation in RL. We first introduce two efficient algorithms for replicable random design regression and uncentered covariance estimation, each of independent interest. We then leverage these tools to provide the first provably efficient replicable RL algorithms for linear Markov decision processes in both the generative model and episodic settings. Finally, we evaluate our algorithms experimentally and show how they can inspire more consistent neural policies.

Replicable Reinforcement Learning with Linear Function Approximation

TL;DR

This paper tackles the problem of ensuring exact replicability in reinforcement learning when using linear function approximation in linear MDPs. It introduces two replicable tools—replicable ridge regression and replicable uncentered covariance estimation—and builds replicable RL algorithms for both generative-model and episodic settings (R-LSVI with core sets and R-LSVI-UCB) atop these tools. The authors provide finite-sample replicability guarantees and concrete sample complexities, supported by experiments on real datasets and neural Q-value quantization to illustrate practical stability benefits. Overall, the work advances reliable RL by enabling exact repeatability across runs, paving the way for safer and more auditable AI systems while outlining directions for future feature-learning and scalability to non-linear regimes.

Abstract

Replication of experimental results has been a challenge faced by many scientific disciplines, including the field of machine learning. Recent work on the theory of machine learning has formalized replicability as the demand that an algorithm produce identical outcomes when executed twice on different samples from the same distribution. Provably replicable algorithms are especially interesting for reinforcement learning (RL), where algorithms are known to be unstable in practice. While replicable algorithms exist for tabular RL settings, extending these guarantees to more practical function approximation settings has remained an open problem. In this work, we make progress by developing replicable methods for linear function approximation in RL. We first introduce two efficient algorithms for replicable random design regression and uncentered covariance estimation, each of independent interest. We then leverage these tools to provide the first provably efficient replicable RL algorithms for linear Markov decision processes in both the generative model and episodic settings. Finally, we evaluate our algorithms experimentally and show how they can inspire more consistent neural policies.

Paper Structure

This paper contains 30 sections, 20 theorems, 123 equations, 2 figures, 1 table, 5 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Linear Markov decision processes
  4. Replicability
  5. Replicable Tools for Linear Spaces
  6. Replicable ridge regression
  7. Replicable uncentered covariance estimation
  8. Replicable RL with linear function approximation
  9. Replicable linear RL with generative models
  10. Replicable linear RL with exploration
  11. Limitations
  12. Experimental evaluation
  13. Evaluating replicability on real datasets
  14. Quantizing neural Q-values
  15. Related work
  16. ...and 15 more sections

Key Result

Proposition 2.1

For any linear MDP and policy $\pi$, there exists a set of weights $\{\mathbf{w}_{h}^{\pi}\}_{h \in [H]}$ such that for all $(s, a, h) \in \mathcal{S} \times \mathcal{A} \times [H]$, we have $Q_h^\pi(s, a) = \langle \phi(s, a), \mathbf{w}_h^\pi \rangle$ .

Figures (2)

  • Figure 1: Mean return and percentage of most common identical weight vector. "Not replicable" indicates a baseline without regularization and rounding. Using only a fraction of the data is sufficient to achieve replicability.
  • Figure 2: Mean return and agreement on MsPacman (left) Breakout (right). Quantization and regularization increase agreement while maintaining high performance.

Theorems & Definitions (40)

  • Definition 2.1: Linear MDP
  • Proposition 2.1: jin2020provably
  • Proposition 2.2: jin2020provably
  • Definition 2.2: Replicability impagliazzo2022reproducibility
  • Lemma 2.1: adapted from impagliazzo2022reproducibility
  • proof
  • Theorem 3.1
  • Definition 3.1: Core set
  • Theorem 3.2: Fixed Design R-Ridge Regression Error
  • Theorem 3.3
  • ...and 30 more