Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stochastic Approximation Methods for Distortion Risk Measure Optimization

Jinyang Jiang, Bernd Heidergott, Jiaqiao Hu, Yijie Peng

TL;DR

This work addresses optimization under risk using Distortion Risk Measures (DRMs) by developing gradient-based stochastic approximation methods grounded in two dual representations: the Distortion-Measure (DM-form) and Quantile-Function (QF-form). It introduces a three-timescale SA for the DM-form with kernel density estimation and GLR-based gradient estimators, and a two-timescale SA for the QF-form that avoids density estimation, plus a Hybrid that combines both. Theoretical contributions include strong convergence proofs and finite-sample MSE rates, with $O(k^{-4/7})$ for the DM-form and $O(k^{-2/3})$ for the QF-form, along with decoupled integration and SA errors. Numerical results on robust portfolio optimization and DRM-enabled deep RL (DPPO) demonstrate significant performance gains and scalability, including a multi-echelon inventory example. The work provides practical, scalable DRM optimization tools applicable to finance and sequential decision-making under uncertainty, with accompanying code release.

Abstract

Distortion Risk Measures (DRMs) capture risk preferences in decision-making and serve as general criteria for managing uncertainty. This paper proposes gradient descent algorithms for DRM optimization based on two dual representations: the Distortion-Measure (DM) form and Quantile-Function (QF) form. The DM-form employs a three-timescale algorithm to track quantiles, compute their gradients, and update decision variables, utilizing the Generalized Likelihood Ratio and kernel-based density estimation. The QF-form provides a simpler two-timescale approach that avoids the need for complex quantile gradient estimation. A hybrid form integrates both approaches, applying the DM-form for robust performance around distortion function jumps and the QF-form for efficiency in smooth regions. Proofs of strong convergence and convergence rates for the proposed algorithms are provided. In particular, the DM-form achieves an optimal rate of $O(k^{-4/7})$, while the QF-form attains a faster rate of $O(k^{-2/3})$. Numerical experiments confirm their effectiveness and demonstrate substantial improvements over baselines in robust portfolio selection tasks. The method's scalability is further illustrated through integration into deep reinforcement learning. Specifically, a DRM-based Proximal Policy Optimization algorithm is developed and applied to multi-echelon dynamic inventory management, showcasing its practical applicability.

Stochastic Approximation Methods for Distortion Risk Measure Optimization

TL;DR

This work addresses optimization under risk using Distortion Risk Measures (DRMs) by developing gradient-based stochastic approximation methods grounded in two dual representations: the Distortion-Measure (DM-form) and Quantile-Function (QF-form). It introduces a three-timescale SA for the DM-form with kernel density estimation and GLR-based gradient estimators, and a two-timescale SA for the QF-form that avoids density estimation, plus a Hybrid that combines both. Theoretical contributions include strong convergence proofs and finite-sample MSE rates, with for the DM-form and for the QF-form, along with decoupled integration and SA errors. Numerical results on robust portfolio optimization and DRM-enabled deep RL (DPPO) demonstrate significant performance gains and scalability, including a multi-echelon inventory example. The work provides practical, scalable DRM optimization tools applicable to finance and sequential decision-making under uncertainty, with accompanying code release.

Abstract

Distortion Risk Measures (DRMs) capture risk preferences in decision-making and serve as general criteria for managing uncertainty. This paper proposes gradient descent algorithms for DRM optimization based on two dual representations: the Distortion-Measure (DM) form and Quantile-Function (QF) form. The DM-form employs a three-timescale algorithm to track quantiles, compute their gradients, and update decision variables, utilizing the Generalized Likelihood Ratio and kernel-based density estimation. The QF-form provides a simpler two-timescale approach that avoids the need for complex quantile gradient estimation. A hybrid form integrates both approaches, applying the DM-form for robust performance around distortion function jumps and the QF-form for efficiency in smooth regions. Proofs of strong convergence and convergence rates for the proposed algorithms are provided. In particular, the DM-form achieves an optimal rate of , while the QF-form attains a faster rate of . Numerical experiments confirm their effectiveness and demonstrate substantial improvements over baselines in robust portfolio selection tasks. The method's scalability is further illustrated through integration into deep reinforcement learning. Specifically, a DRM-based Proximal Policy Optimization algorithm is developed and applied to multi-echelon dynamic inventory management, showcasing its practical applicability.

Paper Structure

This paper contains 28 sections, 18 theorems, 96 equations, 3 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation & Preliminaries
  3. Related Literature
  4. The DM-Form: Three-Timescale DRM Optimization
  5. The QF-Form: Two-Timescale DRM Optimization
  6. The Hybrid Form
  7. Numerical Results
  8. Robust Portfolio Selection with DRM Criteria
  9. Risk-sensitive Inventory Management via DRM-based Deep RL
  10. Conclusions
  11. Gradient Expressions
  12. Proofs of Theorems
  13. Auxiliary Results
  14. Proof of Theorem \ref{['th:main_convergence']}
  15. Proof of Theorem \ref{['th:mse_q']}
  16. ...and 13 more sections

Key Result

Theorem 1

If Assumptions aspt:diff_quantile-aspt:density hold, then the sequence $\{\theta_k\}_{k\in\mathbb{N}}$ generated by recursions (eq:iter_d)-(eq:iter_theta) converge a.s. to some limit set of the ODE (eq:ode_final). More formally, it holds w.p.1 that $\lim_{t \rightarrow \infty } {\theta}(t) =\lim_{k

Figures (3)

  • Figure 1: Visualization of example distortion functions and their derivatives.
  • Figure 2: Visualization of the optimization procedure under different distortion functions. The top row shows DRM learning curves, and the bottom row shows the Wasserstein-2 distances between fitted and optimal quantile functions.
  • Figure 3: Performance visualization of PPO and DPPO variants in the inventory management example.

Theorems & Definitions (21)

  • Remark 1
  • Remark 2
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Remark 3
  • Theorem 5
  • Theorem 6
  • Theorem 7
  • ...and 11 more