Explainable Post hoc Portfolio Management Financial Policy of a Deep Reinforcement Learning agent

TL;DR

The paper tackles the need for interpretable decision-making in DRL-driven portfolio management under volatility by proposing an Explainable post-hoc DRL (XDRL) framework that pairs PPO with model-agnostic explanations (SHAP, LIME, and feature importance). It demonstrates prediction-time interpretability of trading actions using a Yahoo Finance OHCLV data pipeline and a PPO-based agent trained on historical data, showing that key features (e.g., Apple’s metrics) drive asset allocations in concrete, explainable ways. The contributions include establishing a first-of-its-kind post-hoc explainable policy for DRL portfolio management, and validating the approach through qualitative explanation visualizations and local/global feature analyses. The work has practical implications for transparency, regulatory compliance, and investor trust in automated portfolio strategies, while outlining avenues for broader market testing and human-in-the-loop enhancements.

Abstract

Financial portfolio management investment policies computed quantitatively by modern portfolio theory techniques like the Markowitz model rely on a set on assumptions that are not supported by data in high volatility markets. Hence, quantitative researchers are looking for alternative models to tackle this problem. Concretely, portfolio management is a problem that has been successfully addressed recently by Deep Reinforcement Learning (DRL) approaches. In particular, DRL algorithms train an agent by estimating the distribution of the expected reward of every action performed by an agent given any financial state in a simulator. However, these methods rely on Deep Neural Networks model to represent such a distribution, that although they are universal approximator models, they cannot explain its behaviour, given by a set of parameters that are not interpretable. Critically, financial investors policies require predictions to be interpretable, so DRL agents are not suited to follow a particular policy or explain their actions. In this work, we developed a novel Explainable Deep Reinforcement Learning (XDRL) approach for portfolio management, integrating the Proximal Policy Optimization (PPO) with the model agnostic explainable techniques of feature importance, SHAP and LIME to enhance transparency in prediction time. By executing our methodology, we can interpret in prediction time the actions of the agent to assess whether they follow the requisites of an investment policy or to assess the risk of following the agent suggestions. To the best of our knowledge, our proposed approach is the first explainable post hoc portfolio management financial policy of a DRL agent. We empirically illustrate our methodology by successfully identifying key features influencing investment decisions, which demonstrate the ability to explain the agent actions in prediction time.

Figures (8)

• Figure 1: Deep reinforcement learning main components that enable the estimation of the expected reward of any action of the action space of the agent conditioned to any state perceived by the agent. The learnt policy function is encoded by a deep neural network, enabling continuous-valued actions and spaces and any complexity of its mapping.
• Figure 2: Importance of the features used as the state space of the DRL agent for financial portfolio management experiments. We can see how the APPLE close value is the most important for the estimated policy of the DRL agent.
• Figure 3: Feature importance of the state space of the DRL agent sorted by assets.
• Figure 4: Mean feature importance of the different financial indicators across all the assets of the portfolio.
• Figure 5: SHAP force plot for AAPL weight allocation with all features contribution.