Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Exploratory Stopping under Ambiguity in Reinforcement Learning

Junyan Ye, Hoi Ying Wong, Kyunghyun Park

TL;DR

This work develops a robust, continuous-time reinforcement learning framework for optimal stopping under ambiguity, leveraging the $g$-expectation and backward SDEs to model model uncertainty. By introducing entropy-regularized exploration with Bernoulli-like stopping controls and a logistic update, it derives an explicit optimal exploratory control and proves a policy-iteration scheme that converges to the robust solution. The approach connects stochastic control with RL via penalized and reflected BSDEs, and provides a Markovian PDE representation suitable for numerical implementation through the deep-splitting method. Numerical experiments on American put/call problems corroborate convergence, robustness to ambiguity levels, and stable performance under distributional shifts. Overall, the framework offers a principled way to balance exploration and robust learning in continuous-time stopping problems under model ambiguity.

Abstract

We propose and analyze a continuous-time robust reinforcement learning framework for optimal stopping problems under ambiguity. In this framework, an agent chooses a stopping rule motivated by two objectives: robust decision-making under ambiguity and learning about the unknown environment. Here, ambiguity refers to considering multiple probability measures dominated by a reference measure, reflecting the agent's awareness that the reference measure representing her learned belief about the environment would be erroneous. Using the $g$-expectation framework, we reformulate an optimal stopping problem under ambiguity as an entropy-regularized optimal control problem under ambiguity, with Bernoulli distributed controls to incorporate exploration into the stopping rules. We then derive the optimal Bernoulli distributed control characterized by backward stochastic differential equations. Moreover, we establish a policy iteration theorem and implement it as a reinforcement learning algorithm. Numerical experiments demonstrate the convergence and robustness of the proposed algorithm across different levels of ambiguity and exploration.

Robust Exploratory Stopping under Ambiguity in Reinforcement Learning

TL;DR

This work develops a robust, continuous-time reinforcement learning framework for optimal stopping under ambiguity, leveraging the -expectation and backward SDEs to model model uncertainty. By introducing entropy-regularized exploration with Bernoulli-like stopping controls and a logistic update, it derives an explicit optimal exploratory control and proves a policy-iteration scheme that converges to the robust solution. The approach connects stochastic control with RL via penalized and reflected BSDEs, and provides a Markovian PDE representation suitable for numerical implementation through the deep-splitting method. Numerical experiments on American put/call problems corroborate convergence, robustness to ambiguity levels, and stable performance under distributional shifts. Overall, the framework offers a principled way to balance exploration and robust learning in continuous-time stopping problems under model ambiguity.

Abstract

We propose and analyze a continuous-time robust reinforcement learning framework for optimal stopping problems under ambiguity. In this framework, an agent chooses a stopping rule motivated by two objectives: robust decision-making under ambiguity and learning about the unknown environment. Here, ambiguity refers to considering multiple probability measures dominated by a reference measure, reflecting the agent's awareness that the reference measure representing her learned belief about the environment would be erroneous. Using the -expectation framework, we reformulate an optimal stopping problem under ambiguity as an entropy-regularized optimal control problem under ambiguity, with Bernoulli distributed controls to incorporate exploration into the stopping rules. We then derive the optimal Bernoulli distributed control characterized by backward stochastic differential equations. Moreover, we establish a policy iteration theorem and implement it as a reinforcement learning algorithm. Numerical experiments demonstrate the convergence and robustness of the proposed algorithm across different levels of ambiguity and exploration.

Paper Structure

This paper contains 11 sections, 8 theorems, 77 equations, 2 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related literature
  3. Notations and preliminaries
  4. Optimal stopping under ambiguity
  5. Exploratory framework: approximation of optimal stopping under ambiguity
  6. Policy iteration theorem & RL algorithm
  7. Experiments
  8. Proofs
  9. Proof of results in Section \ref{['sec:robust_stopping']}
  10. Proof of results in Section \ref{['subsec:robust_exploratory']}
  11. Proof of results in Section \ref{['sec:policy_iter']}

Key Result

Proposition 2.10

Suppose that Assumptions as:refer_semi and as:reward hold. Let $(V_t^x)_{t\in[0,T]}$ be given in eq:robust_stopping (see Remark rem:wellposed_stopping) and let $(Y_t^x)_{t\in[0,T]}$ be the first component of the unique solution to the reflected BSDE eq:reflected_gBSDE with the lower obstacle $(R(X_t is optimal to the robust stopping problem $V^x$.

Figures (2)

  • Figure 1: Policy improvement and convergence in Algorithm \ref{['alg:exact_policy_iteration']} under several ambiguity levels.
  • Figure 2: Robustness performance under unknown testing environments.

Theorems & Definitions (29)

  • Definition 2.1
  • Remark 2.2
  • Remark 2.4
  • Remark 2.5
  • Remark 2.7
  • Definition 2.8
  • Remark 2.9
  • Proposition 2.10
  • Remark 2.11
  • Proposition 2.12
  • ...and 19 more