Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Pontryagin-Guided Policy Optimization for Merton's Portfolio Problem

Jeonggyu Huh, Jaegi Jeon

Abstract

We present a Pontryagin-Guided Direct Policy Optimization (PG-DPO) framework for Merton's portfolio problem, unifying modern neural-network-based policy parameterization with the adjoint viewpoint from Pontryagin's maximum principle (PMP). Instead of approximating the value function (as done in deep BSDE methods), we track a policy-fixed BSDE for the adjoint processes, which allows each gradient update to align with continuous-time PMP conditions. This setup yields locally optimal consumption and investment policies that are closely tied to classical stochastic control. We further incorporate an alignment penalty that nudges the learned policy toward Pontryagin-derived solutions, enhancing both convergence speed and training stability. Numerical experiments confirm that PG-DPO effectively handles both consumption and investment, achieving strong performance and interpretability without requiring large offline datasets or model-free reinforcement learning.

Pontryagin-Guided Policy Optimization for Merton's Portfolio Problem

Abstract

We present a Pontryagin-Guided Direct Policy Optimization (PG-DPO) framework for Merton's portfolio problem, unifying modern neural-network-based policy parameterization with the adjoint viewpoint from Pontryagin's maximum principle (PMP). Instead of approximating the value function (as done in deep BSDE methods), we track a policy-fixed BSDE for the adjoint processes, which allows each gradient update to align with continuous-time PMP conditions. This setup yields locally optimal consumption and investment policies that are closely tied to classical stochastic control. We further incorporate an alignment penalty that nudges the learned policy toward Pontryagin-derived solutions, enhancing both convergence speed and training stability. Numerical experiments confirm that PG-DPO effectively handles both consumption and investment, achieving strong performance and interpretability without requiring large offline datasets or model-free reinforcement learning.

Paper Structure

This paper contains 31 sections, 2 theorems, 58 equations, 1 figure, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Continuous-Time Formulation of the Merton Portfolio Problem
  3. Pontryagin's Maximum Principle and Its Application to Merton's Problem
  4. Pontryagin's Maximum Principle
  5. Pontryagin Adjoint Processes and Parameter Gradients in the Merton Problem
  6. Pontryagin Formulation for Merton's Problem
  7. Remark (Expressing $\pi_{t}^{*}$ in terms of $\lambda_{t}^{*}$ and $\,Z_{t}^{*}$ under regularity).
  8. Policy-Fixed Adjoint Processes for Suboptimal Policies
  9. Iterative Policy Improvement
  10. Gradient with respect to $\theta$.
  11. Gradient with respect to $\phi$.
  12. Deriving \ref{['eq:generic_chainrule']} via the Chain Rule.
  13. Comparison with Deep BSDE Methods
  14. Value-Based Methods (Deep BSDE)
  15. Hamilton--Jacobi--Bellman Equation and the Value Function
  16. ...and 16 more sections

Key Result

Theorem 1

Suppose (A1)--(A3) hold, and that $\|\delta_{k}\|\to0$ as $\Delta t\to0$. Then, with probability one, where $\nabla J(\theta^{\dagger},\phi^{\dagger})=0$. In other words, $(\theta^{\dagger},\phi^{\dagger})$ is a stationary point of $J$ in the parameter space.

Figures (1)

  • Figure 1: Neural vs. exact solutions for consumption/investment under the Merton model. The learned plots (left) are from our PG-DPO-Align run at iteration 100,000; the exact plots (right) show the closed-form Merton solution.

Theorems & Definitions (4)

  • Theorem 1: Baseline Robbins--Monro Convergence
  • proof : Proof Sketch
  • Theorem 2: Stationarity of $\widetilde{J}$
  • proof : Proof Sketch