Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UAMDP: Uncertainty-Aware Markov Decision Process for Risk-Constrained Reinforcement Learning from Probabilistic Forecasts

Michal Koren, Or Peretz, Tai Dinh, Philip S. Yu

TL;DR

The paper introduces UAMDP, a Bayes-adaptive framework that tightly couples probabilistic forecasting with posterior-sampling reinforcement learning and CVaR-based risk constraints to enable uncertainty-aware sequential decision-making. By maintaining a posterior over latent dynamics, sampling plausible futures, and planning under a risk-aware objective, UAMDP improves both forecast calibration and downstream control performance in high-stakes domains. Empirical results in high-frequency equity trading and retail inventory demonstrate substantial gains in forecast accuracy, risk-adjusted returns, and robustness to perturbations, outperforming strong baselines. This work provides a practical blueprint for integrating calibrated probabilistic forecasts into closed-loop control with principled tail-risk management, offering broad implications for safer and more profitable decision systems in volatile environments.

Abstract

Sequential decisions in volatile, high-stakes settings require more than maximizing expected return; they require principled uncertainty management. This paper presents the Uncertainty-Aware Markov Decision Process (UAMDP), a unified framework that couples Bayesian forecasting, posterior-sampling reinforcement learning, and planning under a conditional value-at-risk (CVaR) constraint. In a closed loop, the agent updates its beliefs over latent dynamics, samples plausible futures via Thompson sampling, and optimizes policies subject to preset risk tolerances. We establish regret bounds that converge to the Bayes-optimal benchmark under standard regularity conditions. We evaluate UAMDP in two domains including high-frequency equity trading and retail inventory control, both marked by structural uncertainty and economic volatility. Relative to strong deep learning baselines, UAMDP improves long-horizon forecasting accuracy (RMSE decreases by up to 25% and sMAPE by 32%), and these gains translate into economic performance: the trading Sharpe ratio rises from 1.54 to 1.74 while maximum drawdown is roughly halved. These results show that integrating calibrated probabilistic modeling, exploration aligned with posterior uncertainty, and risk-aware control yields a robust, generalizable approach to safer and more profitable sequential decision-making.

UAMDP: Uncertainty-Aware Markov Decision Process for Risk-Constrained Reinforcement Learning from Probabilistic Forecasts

TL;DR

The paper introduces UAMDP, a Bayes-adaptive framework that tightly couples probabilistic forecasting with posterior-sampling reinforcement learning and CVaR-based risk constraints to enable uncertainty-aware sequential decision-making. By maintaining a posterior over latent dynamics, sampling plausible futures, and planning under a risk-aware objective, UAMDP improves both forecast calibration and downstream control performance in high-stakes domains. Empirical results in high-frequency equity trading and retail inventory demonstrate substantial gains in forecast accuracy, risk-adjusted returns, and robustness to perturbations, outperforming strong baselines. This work provides a practical blueprint for integrating calibrated probabilistic forecasts into closed-loop control with principled tail-risk management, offering broad implications for safer and more profitable decision systems in volatile environments.

Abstract

Sequential decisions in volatile, high-stakes settings require more than maximizing expected return; they require principled uncertainty management. This paper presents the Uncertainty-Aware Markov Decision Process (UAMDP), a unified framework that couples Bayesian forecasting, posterior-sampling reinforcement learning, and planning under a conditional value-at-risk (CVaR) constraint. In a closed loop, the agent updates its beliefs over latent dynamics, samples plausible futures via Thompson sampling, and optimizes policies subject to preset risk tolerances. We establish regret bounds that converge to the Bayes-optimal benchmark under standard regularity conditions. We evaluate UAMDP in two domains including high-frequency equity trading and retail inventory control, both marked by structural uncertainty and economic volatility. Relative to strong deep learning baselines, UAMDP improves long-horizon forecasting accuracy (RMSE decreases by up to 25% and sMAPE by 32%), and these gains translate into economic performance: the trading Sharpe ratio rises from 1.54 to 1.74 while maximum drawdown is roughly halved. These results show that integrating calibrated probabilistic modeling, exploration aligned with posterior uncertainty, and risk-aware control yields a robust, generalizable approach to safer and more profitable sequential decision-making.

Paper Structure

This paper contains 25 sections, 4 theorems, 42 equations, 9 figures, 13 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Probabilistic time-series forecasting
  4. Reinforcement learning under uncertainty
  5. Integrated forecasting and control
  6. Uncertainty-aware Markov decision process
  7. Procedure flow
  8. Correctness
  9. Assumption
  10. Preliminaries
  11. Proof of correctness
  12. Error bounds
  13. Empirical study
  14. Data sources
  15. Pre-processing and feature engineering
  16. ...and 10 more sections

Key Result

Lemma 1

Under any policy $\pi \in \Pi$, the sequence $\{(b_t,s_t)\}_{t\ge 0}$ is a Markov chain with kernel $\tilde{p}$.

Figures (9)

  • Figure 1: Flow of the proposed framework.
  • Figure 2: Fan chart of 50 posterior sample paths for a representative equity segment (left) and garment family (right).
  • Figure 3: Reliability diagrams for all models.
  • Figure 4: Cumulative profit and loss for S&P 500 portfolio.
  • Figure 5: Holding cost vs. stockout tradeoff for UAMDP and TFT.
  • ...and 4 more figures

Theorems & Definitions (4)

  • Lemma 1: Belief Markov Property
  • Lemma 2: MDP Optimality
  • Theorem 1: Bayes Optimality under Exact Inference
  • Theorem 2: Finite-time regret under approximate inference