Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data-Driven Merton's Strategies via Policy Randomization

Min Dai, Yuchao Dong, Yanwei Jia, Xun Yu Zhou

TL;DR

This paper introduces a data-driven solution to Merton’s portfolio problem in incomplete markets by embedding the problem in a continuous-time RL framework with Gaussian policy randomization. It proves that the mean of the optimal Gaussian policy for an auxiliary stochastic-control problem coincides with the true Merton optimal policy, enabling model-free learning of both the policy and value functions via online and offline actor–critic methods. The authors develop theory (policy-improvement, martingale-orthogonality) and practical algorithms, and demonstrate robustness and superior performance relative to model-based plug-in methods through both synthetic simulations in stochastic volatility settings and an empirical study using real market data. The approach highlights that policy randomization serves not only exploration but also a tractable route to solving otherwise intractable stochastic control problems when primitives are unknown. The results suggest RL-based portfolio strategies can achieve strong risk-adjusted performance with improved robustness to parameter misspecification and observational noise. The work bridges continuous-time finance and modern RL, offering a scalable, data-driven paradigm for dynamic investment under uncertainty.

Abstract

We study Merton's expected utility maximization problem in an incomplete market, characterized by a factor process in addition to the stock price process, where all the model primitives are unknown. The agent under consideration is a price taker who has access only to the stock and factor value processes and the instantaneous volatility. We propose an auxiliary problem in which the agent can invoke policy randomization according to a specific class of Gaussian distributions, and prove that the mean of its optimal Gaussian policy solves the original Merton problem. With randomized policies, we are in the realm of continuous-time reinforcement learning (RL) recently developed in Wang et al. (2020) and Jia and Zhou (2022a, 2022b, 2023), enabling us to solve the auxiliary problem in a data-driven way without having to estimate the model primitives. Specifically, we establish a policy improvement theorem based on which we design both online and offline actor-critic RL algorithms for learning Merton's strategies. A key insight from this study is that RL in general and policy randomization in particular are useful beyond the purpose for exploration -- they can be employed as a technical tool to solve a problem that cannot be otherwise solved by mere deterministic policies. At last, we carry out both simulation and empirical studies in a stochastic volatility environment to demonstrate the decisive outperformance of the devised RL algorithms in comparison to the conventional model-based, plug-in method.

Data-Driven Merton's Strategies via Policy Randomization

TL;DR

This paper introduces a data-driven solution to Merton’s portfolio problem in incomplete markets by embedding the problem in a continuous-time RL framework with Gaussian policy randomization. It proves that the mean of the optimal Gaussian policy for an auxiliary stochastic-control problem coincides with the true Merton optimal policy, enabling model-free learning of both the policy and value functions via online and offline actor–critic methods. The authors develop theory (policy-improvement, martingale-orthogonality) and practical algorithms, and demonstrate robustness and superior performance relative to model-based plug-in methods through both synthetic simulations in stochastic volatility settings and an empirical study using real market data. The approach highlights that policy randomization serves not only exploration but also a tractable route to solving otherwise intractable stochastic control problems when primitives are unknown. The results suggest RL-based portfolio strategies can achieve strong risk-adjusted performance with improved robustness to parameter misspecification and observational noise. The work bridges continuous-time finance and modern RL, offering a scalable, data-driven paradigm for dynamic investment under uncertainty.

Abstract

We study Merton's expected utility maximization problem in an incomplete market, characterized by a factor process in addition to the stock price process, where all the model primitives are unknown. The agent under consideration is a price taker who has access only to the stock and factor value processes and the instantaneous volatility. We propose an auxiliary problem in which the agent can invoke policy randomization according to a specific class of Gaussian distributions, and prove that the mean of its optimal Gaussian policy solves the original Merton problem. With randomized policies, we are in the realm of continuous-time reinforcement learning (RL) recently developed in Wang et al. (2020) and Jia and Zhou (2022a, 2022b, 2023), enabling us to solve the auxiliary problem in a data-driven way without having to estimate the model primitives. Specifically, we establish a policy improvement theorem based on which we design both online and offline actor-critic RL algorithms for learning Merton's strategies. A key insight from this study is that RL in general and policy randomization in particular are useful beyond the purpose for exploration -- they can be employed as a technical tool to solve a problem that cannot be otherwise solved by mere deterministic policies. At last, we carry out both simulation and empirical studies in a stochastic volatility environment to demonstrate the decisive outperformance of the devised RL algorithms in comparison to the conventional model-based, plug-in method.
Paper Structure (30 sections, 9 theorems, 96 equations, 5 figures, 2 tables, 2 algorithms)

This paper contains 30 sections, 9 theorems, 96 equations, 5 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Market Environment and Investment Objective
  4. Agent's Knowledge and Randomized Choices
  5. An Auxiliary Problem with Gaussian Policies
  6. Theoretical Analysis
  7. Ground Truth Solution to the Auxiliary Problem
  8. Reinforcement Learning Methods for Solving the Auxiliary Problem
  9. Data-Driven RL Algorithms
  10. An Example: The Black--Scholes Market
  11. An informal analysis: optimality conditions.
  12. A formal analysis: impacts of discretization and randomization.
  13. A Market with Stochastic Volatility
  14. Classical Benchmark
  15. Pitfall of Model-Based Solution and Virtue of Reinforcement Learning
  16. ...and 15 more sections

Key Result

Theorem 1

Suppose $\varphi$ is a classical solution of the following PDE with the terminal condition $\varphi(T,x) = 0$, and $\varphi$ satisfies the regularity condition that $\{ e^{ (1+\epsilon) \varphi(t,X_t) } \}_{t\in [0,T]}$ is uniformly integrable for some $\epsilon > 0$. Then is a classical solution to the HJB equation eq:hjb 0. Moreover, is the optimal policy to the auxiliary problem optimalvalue

Figures (5)

  • Figure 1: The convergence of the learned policy (in terms of ERWL) under different temperature parameters $\lambda$. The horizontal (the number of episodes) and vertical (expected relative wealth loss) axes are both in log-scale. The shaded areas indicate the standard deviations of the estimated ERWLs. The results are based on 1000 times of independent simulation runs and 10,000 episodes of 1-year trajectory is used in each run. The model parameters are $\mu = 0.2,r=0.02,\sigma=0.3,\gamma=3,T=1$. The learning rate is $a_n = 10/(n+1)$ and the initial policy parameter is $\theta_0 = 0$. The projected region is taken as $c_n = \max\{10, \sqrt{\log (n+1)} \}$ and discretization size is $\Delta t_n = \min\{ 0.001, 10/(n+1) \}$.
  • Figure 2: Wealth trajectories of portfolios under different policies. The gray plot is the VIX index whose vertical axis is on the right. The other plots are the trajectories of the portfolio values under different methods and are all normalized to 1 initially.
  • Figure 3: Trajectories of risky proportions under different policies. In our study, proportions invested in S&P 500 are restricted to be between 0 and 1. The gray curve is the VIX index whose vertical axis is on the right. The other curves are the trajectories of the proportions of the risky investment under different methods. The initial allocations of all the methods (except B-H) are based on the pre-training period from 1990 to 1999.
  • Figure 4: The comparison between the empirical risk minimization and the proposed method. The horizontal (the number of episodes) and vertical (expected relative wealth loss) axes are both in log-scale. The shaded areas indicate the standard deviations of the estimated ERWLs. The results are based on 1000 times of independent simulation runs and 10,000 episodes of 1-year trajectory is used in each run. The model parameters are $\mu = 0.2,r=0.02,\sigma=0.3,\gamma=3,T=1$. The learning rate is $a_n = 10/(n+1)$ and the initial policy parameter is $\theta_0 = 0$. The projected region is taken as $c_n = \max\{10, \sqrt{\log (n+1)} \}$ and discretization size is $\Delta t_n = \min\{ 0.001, 10/(n+1) \}$.
  • Figure 5: Average utility of RL algorithms on the test set as functions of the number of training episodes. In each episode, independent one-year data are generated from the model dynamics for training. The width of the shaded area is twice the standard deviation.

Theorems & Definitions (11)

  • Definition 1
  • Theorem 1
  • Definition 2
  • Corollary 1
  • Theorem 2
  • Theorem 3
  • Proposition 1
  • Theorem 4
  • Lemma 1
  • Lemma 2
  • ...and 1 more