Mean--Variance Portfolio Selection by Continuous-Time Reinforcement Learning: Algorithms, Regret Analysis, and Empirical Study

Abstract

We study continuous-time mean--variance portfolio selection in markets where stock prices are diffusion processes driven by observable factors that are also diffusion processes, yet the coefficients of these processes are unknown. Based on the recently developed reinforcement learning (RL) theory for diffusion processes, we present a general data-driven RL approach that learns the pre-committed investment strategy directly without attempting to learn or estimate the market coefficients. For multi-stock Black--Scholes markets without factors, we further devise an algorithm and prove its performance guarantee by deriving a sublinear regret bound in terms of the Sharpe ratio. We then carry out an extensive empirical study implementing this algorithm to compare its performance and trading characteristics, evaluated under a host of common metrics, with a large number of widely employed portfolio allocation strategies on S\&P 500 constituents. The results demonstrate that the proposed continuous-time RL strategy is consistently among the best, especially in a volatile bear market, and decisively outperforms the model-based continuous-time counterparts by significant margins.

Figures (8)

• Figure 1: Wealth trajectories under the proposed CTRL algorithm and 13 alternative methods over 100 independent experiments, each with 10 randomly selected stocks (except the S&P 500 index), from 2000 to 2020. The solid line shows the median across experiments at each time, and the shaded region is the interquartile range.
• Figure 2: Error of parameter $\phi_1$ The solid curves and the upper and lower boundaries of the shaded regions represent the average, 2.5% and 97.5% percentile of the error over 1000 independent simulation runs, respectively. The slope for $\phi_1$ by least squares regression is -1.09. The vertical and horizontal axes are on natural log-scale.
• Figure 3: Error of parameter $\phi_2$ The solid curves and the upper and lower boundaries of the shaded regions represent the average, 2.5% and 97.5% percentile of the error over 1000 independent simulation runs, respectively. The slope for $\phi_2$ by least squares regression is -0.91. The vertical and horizontal axes are on natural log-scale.
• Figure 4: Error of parameter $w$ The solid curves and the upper and lower boundaries of the shaded regions represent the average, 2.5% and 97.5% percentile of the error over 1000 independent simulation runs, respectively. The slope for $w$ by least squares regression is -0.97. The vertical and horizontal axes are on natural log-scale.
• Figure 5: Cumulative regret rate in number of episodes. The solid blue curve and the upper and lower boundary of the shaded region represent the mean, 2.5% and 97.5% percentile of the regret over 1000 independent simulation runs, respectively. The red dashed line is the fitted value by linearly regressing the log average regret against the logarithm of the number of episodes starting from the 200th episode. The fitted slope by least squares regression is $0.520$. The vertical and horizontal axes are on natural log-scale.

Theorems & Definitions (12)

• Theorem 1
• Theorem 2
• Theorem 3
• Theorem 4
• Lemma 1
• Lemma 2
• Theorem 5
• Remark 1
• Lemma 3
• Theorem 6