Risk-Aware Deep Reinforcement Learning for Dynamic Portfolio Optimization

TL;DR

This work tackles dynamic portfolio optimization under market uncertainty by integrating a Sharpe ratio–based reward with explicit risk controls within a PPO-based DRL framework. The approach maps market states to asset allocations via a neural policy while enforcing long-only constraints and transaction-cost-aware rewards, aiming to maximize risk-adjusted performance. Empirical results reveal a strong risk reduction after training but a substantial drop in absolute and risk-adjusted returns, highlighting challenges in reward shaping, exploration–exploitation balance, and non-stationarity. The study underscores the need for hybrid risk-aware strategies and robust validation to translate promising pre-training results into practical, stable deployment for DRL-driven portfolio management.

Abstract

This paper presents a deep reinforcement learning (DRL) framework for dynamic portfolio optimization under market uncertainty and risk. The proposed model integrates a Sharpe ratio-based reward function with direct risk control mechanisms, including maximum drawdown and volatility constraints. Proximal Policy Optimization (PPO) is employed to learn adaptive asset allocation strategies over historical financial time series. Model performance is benchmarked against mean-variance and equal-weight portfolio strategies using backtesting on high-performing equities. Results indicate that the DRL agent stabilizes volatility successfully but suffers from degraded risk-adjusted returns due to over-conservative policy convergence, highlighting the challenge of balancing exploration, return maximization, and risk mitigation. The study underscores the need for improved reward shaping and hybrid risk-aware strategies to enhance the practical deployment of DRL-based portfolio allocation models.

Figures (4)

• Figure 1: Efficient Frontier illustrating the trade-off between expected return and risk. Portfolios on the frontier dominate inferior portfolios in terms of return per unit risk. The Capital Market Line (CML) shows the optimal portfolio combinations including a risk-free asset.
• Figure 2: Basic representation of a Deep Reinforcement Learning (DRL) framework. The agent observes the state from the environment and uses a deep neural network (DNN) to compute a policy $\pi(s, a)$. Based on the policy, the agent takes an action, receives a reward, and updates its policy to improve future decisions.
• Figure 3: Comparison between biological neural networks and deep learning architectures. The biological neuron (top left) is the inspiration for artificial neurons in deep learning (bottom left). Signals in biological networks are passed between neurons (top right), similar to how inputs are processed through layers of a deep neural network (bottom right).
• Figure 4: The figure above shows the weight allocation for the equal weights portfolio and the DRL portfolio, which rebalances the weights at each time step or when it iterates using a policy that maximizes the portfolio’s cumulative value.