Time-Inhomogeneous Volatility Aversion for Financial Applications of Reinforcement Learning
Federico Cacciamani, Roberto Daluiso, Marco Pinciroli, Michele Trapletti, Edoardo Vittori
TL;DR
The paper addresses the limitation of traditional RL, which optimizes expected return, for financial sequential decision problems where the time distribution of profits and losses matters. It introduces a time-inhomogeneous risk objective by penalizing deviations from stepwise conditional means, via $\mathbb{E}_\pi[\mathcal{G}] - \beta\varsigma^2_\pi$, and generalizes with inhomogeneous $\ell$-volatility and optimised certainty equivalents, enabling arbitrary per-step targets. The authors develop policy-gradient–based algorithms (IVE/IVO) with a Bellman-like target $X_{\pi,i}$ and derive the gradient of the volatility term, including nested optimization over step targets; they validate the approach on toy environments, a deterministic-horizon optimal execution task, and a stochastic-horizon grid world. The results show that the time-inhomogeneous objective can produce more financially sensible risk-aware policies than homogeneous risk criteria, especially in execution problems, while also providing a flexible framework for further extensions in finance. Overall, this framework enables time-aware risk control in RL for finance, with potential applicability to hedging and budgeting where the timing of rewards is crucial.
Abstract
In finance, sequential decision problems are often faced, for which reinforcement learning (RL) emerges as a promising tool for optimisation without the need of analytical tractability. However, the objective of classical RL is the expected cumulated reward, while financial applications typically require a trade-off between return and risk. In this work, we focus on settings where one cares about the time split of the total return, ruling out most risk-aware generalisations of RL which optimise a risk measure defined on the latter. We notice that a preference for homogeneous splits, which we found satisfactory for hedging, can be unfit for other problems, and therefore propose a new risk metric which still penalises uncertainty of the single rewards, but allows for an arbitrary planning of their target levels. We study the properties of the resulting objective and the generalisation of learning algorithms to optimise it. Finally, we show numerical results on toy examples.