Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Guaranteeing Out-Of-Distribution Detection in Deep RL via Transition Estimation

Mohit Prashant, Arvind Easwaran, Suman Das, Michael Yuhas

TL;DR

This work tackles the reliability of deep RL when deployed in environments that may differ from training by formalizing out-of-distribution (OOD) execution in MDPs. It introduces a transition-focused detector based on an ensemble of CVAEs conditioned on transitions and uses inductive conformal prediction to provide guaranteed detection with a user-specified confidence level. The approach defines two OOD execution modes—state-based and environment-based—and demonstrates, via OpenAI Gym benchmarks, that transition-dynamics modeling with ICP can outperform uncertainty-based detectors, offering higher true-positive guarantees and robust performance across OOD types. The results underscore the practical impact of reliable OOD detection for safer, deployment-ready RL systems and point to future work on trade-offs between detection speed and accuracy, as well as hyperparameter optimization for the CVAE ensemble.

Abstract

An issue concerning the use of deep reinforcement learning (RL) agents is whether they can be trusted to perform reliably when deployed, as training environments may not reflect real-life environments. Anticipating instances outside their training scope, learning-enabled systems are often equipped with out-of-distribution (OOD) detectors that alert when a trained system encounters a state it does not recognize or in which it exhibits uncertainty. There exists limited work conducted on the problem of OOD detection within RL, with prior studies being unable to achieve a consensus on the definition of OOD execution within the context of RL. By framing our problem using a Markov Decision Process, we assume there is a transition distribution mapping each state-action pair to another state with some probability. Based on this, we consider the following definition of OOD execution within RL: A transition is OOD if its probability during real-life deployment differs from the transition distribution encountered during training. As such, we utilize conditional variational autoencoders (CVAE) to approximate the transition dynamics of the training environment and implement a conformity-based detector using reconstruction loss that is able to guarantee OOD detection with a pre-determined confidence level. We evaluate our detector by adapting existing benchmarks and compare it with existing OOD detection models for RL.

Guaranteeing Out-Of-Distribution Detection in Deep RL via Transition Estimation

TL;DR

This work tackles the reliability of deep RL when deployed in environments that may differ from training by formalizing out-of-distribution (OOD) execution in MDPs. It introduces a transition-focused detector based on an ensemble of CVAEs conditioned on transitions and uses inductive conformal prediction to provide guaranteed detection with a user-specified confidence level. The approach defines two OOD execution modes—state-based and environment-based—and demonstrates, via OpenAI Gym benchmarks, that transition-dynamics modeling with ICP can outperform uncertainty-based detectors, offering higher true-positive guarantees and robust performance across OOD types. The results underscore the practical impact of reliable OOD detection for safer, deployment-ready RL systems and point to future work on trade-offs between detection speed and accuracy, as well as hyperparameter optimization for the CVAE ensemble.

Abstract

An issue concerning the use of deep reinforcement learning (RL) agents is whether they can be trusted to perform reliably when deployed, as training environments may not reflect real-life environments. Anticipating instances outside their training scope, learning-enabled systems are often equipped with out-of-distribution (OOD) detectors that alert when a trained system encounters a state it does not recognize or in which it exhibits uncertainty. There exists limited work conducted on the problem of OOD detection within RL, with prior studies being unable to achieve a consensus on the definition of OOD execution within the context of RL. By framing our problem using a Markov Decision Process, we assume there is a transition distribution mapping each state-action pair to another state with some probability. Based on this, we consider the following definition of OOD execution within RL: A transition is OOD if its probability during real-life deployment differs from the transition distribution encountered during training. As such, we utilize conditional variational autoencoders (CVAE) to approximate the transition dynamics of the training environment and implement a conformity-based detector using reconstruction loss that is able to guarantee OOD detection with a pre-determined confidence level. We evaluate our detector by adapting existing benchmarks and compare it with existing OOD detection models for RL.

Paper Structure

This paper contains 22 sections, 4 equations, 1 figure, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Contribution: Defining OOD Execution
  3. Contribution: Guaranteeing OOD Detection
  4. Preliminaries
  5. Markov Decision Processes
  6. Uncertainty in Learning
  7. Epistemic Uncertainty
  8. Aleatoric Uncertainty
  9. Conditional Variational Autoencoders
  10. Inductive Conformal Prediction
  11. Related Works
  12. Uncertainty Measurement
  13. Defining OOD Execution
  14. OOD Detection with VAEs
  15. Defining Out-Of-Distribution in MDPs
  16. ...and 7 more sections

Figures (1)

  • Figure 1: Depicted is the OOD detection architecture, consisting of (green) a trained RL agent and (yellow) the detector consisting of an ensemble of $N$ CVAEs (blue). Denoting the calibration set using $\{ r_1 ... r_{M \times N} \}$, the CVAE ensemble produces $N$ reconstructions of state $s_2$, $\{ s'_1 ... s'_N\}$, conditioned on $s_1$. The reconstruction error with each of $s'_1 ... s'_N$ is denoted using $r'_1 ... r'_N$ and is compared with the elements of the calibration set.