Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning in Real Option Models

Jodi Dianetti, Giorgio Ferrari, Renyuan Xu

TL;DR

An entropy-regularized reinforcement learning approach to optimal stopping problems motivated by real option models is investigated, allowing randomized stopping policies that balance exploitation and exploration and derive an explicit analytical solution to the regularized problem.

Abstract

We investigate an entropy-regularized reinforcement learning (RL) approach to optimal stopping problems motivated by real option models. Classical stopping rules are strict and non-randomized, limiting natural exploration in RL settings. To address this, we introduce entropy regularization, allowing randomized stopping policies that balance exploitation and exploration. We derive an explicit analytical solution to the regularized problem and prove convergence of the associated free boundary to the classical stopping threshold as the entropy vanishes. The regularized problem admits a natural formulation as a singular stochastic control problem. Building on this structure, we propose both model-based and model-free policy iteration algorithms to learn the optimal boundary. The model-free method operates without knowledge of system dynamics, using only trajectories from the stochastic environment. We establish convergence guarantees and illustrate strong numerical performance. This framework provides a principled and tractable approach for data-driven stopping problems under uncertainty.

Reinforcement Learning in Real Option Models

TL;DR

An entropy-regularized reinforcement learning approach to optimal stopping problems motivated by real option models is investigated, allowing randomized stopping policies that balance exploitation and exploration and derive an explicit analytical solution to the regularized problem.

Abstract

We investigate an entropy-regularized reinforcement learning (RL) approach to optimal stopping problems motivated by real option models. Classical stopping rules are strict and non-randomized, limiting natural exploration in RL settings. To address this, we introduce entropy regularization, allowing randomized stopping policies that balance exploitation and exploration. We derive an explicit analytical solution to the regularized problem and prove convergence of the associated free boundary to the classical stopping threshold as the entropy vanishes. The regularized problem admits a natural formulation as a singular stochastic control problem. Building on this structure, we propose both model-based and model-free policy iteration algorithms to learn the optimal boundary. The model-free method operates without knowledge of system dynamics, using only trajectories from the stochastic environment. We establish convergence guarantees and illustrate strong numerical performance. This framework provides a principled and tractable approach for data-driven stopping problems under uncertainty.
Paper Structure (17 sections, 6 theorems, 95 equations, 3 figures, 4 algorithms)

This paper contains 17 sections, 6 theorems, 95 equations, 3 figures, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related literature.
  3. The entropy-regularized real option problem
  4. Exploratory formulation via singular controls
  5. Entropy regularization
  6. The analytical solution to the entropy-regularized real option problem
  7. The reinforcement learning algorithm
  8. Model-based numerical analysis
  9. Model-free implementation
  10. Numerical performance
  11. Detailed Proofs
  12. Proof of Proposition \ref{['prop xi does not explore']}
  13. Proof of Theorem \ref{['theorem verification real option']}
  14. Proof of Theorem \ref{['thm:property_g']}
  15. Proof of Theorem \ref{['prop:policy_improvement']}
  16. ...and 2 more sections

Key Result

Proposition 2.2

For any $x \in \mathbb{R}^n$ we have with $\xi^* := ( \mathds 1 _{ \{ t \geq \tau^* \} } )_t$ and $\tau^*$ optimal for $J(x;\cdot)$. Moreover, $\xi^* := (\mathds 1 _{ \{ t \geq \tau^* \} })_t$ is the unique optimal control for $J^0(x;\cdot).$

Figures (3)

  • Figure 1: Demonstration of the Policy Iteration Algorithm.
  • Figure 2: Exponential initialization. Left: Ground-truth and learned $g$ function in selected iterations. Right: Convergence to the ground truth in $L_1$ norm (outer iterations).
  • Figure 3: Linear initialization. Left: Ground-truth and learned $g$ function in selected iterations . Right: Convergence to the ground truth in $L_1$ norm (outer iterations).

Theorems & Definitions (10)

  • Proposition 2.2
  • Remark 2.3: Non-exploratory behavior of the optimal controls
  • Theorem 3.1: The solution to the entropy-regularized real option problem
  • Theorem 3.2: Vanishing entropy limits
  • Theorem 4.2
  • Theorem 4.3: Policy improvement
  • Remark 4.5: Justification of Assumption \ref{['ass:initial_policy']}
  • Remark 4.6: Examples that satisfy Assumption \ref{['ass:initial_policy']}
  • Theorem 4.7: Policy convergence
  • Remark 4.8