SAMP-HDRL: Segmented Allocation with Momentum-Adjusted Utility for Multi-agent Portfolio Management via Hierarchical Deep Reinforcement Learning

TL;DR

The paper tackles portfolio optimization in non-stationary markets by introducing SAMP-HDRL, a knowledge-driven hierarchical DRL framework that dynamically clusters assets into two groups, uses an upper-level agent to extract global market signals, and employs lower-level masked agents to perform intra-group allocation. A momentum-adjusted exponential utility framework ties together group-level and intra-group decisions with a principled capital-allocation mechanism, further enhanced by a rebound detection module. Empirical results across three market regimes show robust outperformance against nine traditional and nine DRL baselines, with ablations demonstrating the indispensability of upper–lower coordination dynamic clustering and capital allocation. SHAP-based interpretability uncovers a complementary diversified plus concentrated decision pattern across agents, providing transparent insights into hierarchical DRL portfolio decisions. The work advances practical portfolio management by embedding structural market constraints into the DRL pipeline, boosting adaptability, robustness, and interpretability in complex financial environments.

Abstract

Portfolio optimization in non-stationary markets is challenging due to regime shifts, dynamic correlations, and the limited interpretability of deep reinforcement learning (DRL) policies. We propose a Segmented Allocation with Momentum-Adjusted Utility for Multi-agent Portfolio Management via Hierarchical Deep Reinforcement Learning (SAMP-HDRL). The framework first applies dynamic asset grouping to partition the market into high-quality and ordinary subsets. An upper-level agent extracts global market signals, while lower-level agents perform intra-group allocation under mask constraints. A utility-based capital allocation mechanism integrates risky and risk-free assets, ensuring coherent coordination between global and local decisions. backtests across three market regimes (2019--2021) demonstrate that SAMP-HDRL consistently outperforms nine traditional baselines and nine DRL benchmarks under volatile and oscillating conditions. Compared with the strongest baseline, our method achieves at least 5\% higher Return, 5\% higher Sharpe ratio, 5\% higher Sortino ratio, and 2\% higher Omega ratio, with substantially larger gains observed in turbulent markets. Ablation studies confirm that upper--lower coordination, dynamic clustering, and capital allocation are indispensable to robustness. SHAP-based interpretability further reveals a complementary ``diversified + concentrated'' mechanism across agents, providing transparent insights into decision-making. Overall, SAMP-HDRL embeds structural market constraints directly into the DRL pipeline, offering improved adaptability, robustness, and interpretability in complex financial environments.

Figures (11)

• Figure 1: Transaction flow.At the end of time $t-1$ (equivalently, at the beginning of trading period $t$), the system state is described by the price vectors $\boldsymbol{v}^{(1)}_{t-1}$ and $\boldsymbol{v}^{(2)}_{t-1}$, the group values $p^{(1)}_{t-1}$ and $p^{(2)}_{t-1}$, and the allocation weights $\boldsymbol{\omega}^{(1)}_{t-1}$ and $\boldsymbol{\omega}^{(2)}_{t-1}$. The proportional adjustments in group values induced by the rebalancing decision at time $t$ are denoted by $u^{(1)}_t, u^{(2)}_t \in [0,1]$. Immediately after rebalancing, the updated state variables become $\boldsymbol{v}^{(1)\prime}_{t}$, $\boldsymbol{v}^{(2)\prime}_{t}$, $p^{(1)\prime}_{t}$, $p^{(2)\prime}_{t}$, $\boldsymbol{\omega}^{(1)\prime}_{t}$, and $\boldsymbol{\omega}^{(2)\prime}_{t}$, which remain unchanged throughout period $t$. At the end of period $t$, the realized market state is expressed as $\boldsymbol{v}^{(1)}_{t}$, $\boldsymbol{v}^{(2)}_{t}$, $p^{(1)}_{t}$, $p^{(2)}_{t}$, $\boldsymbol{\omega}^{(1)}_{t}$, and $\boldsymbol{\omega}^{(2)}_{t}$. The relative change in asset prices during period $t$ is captured by the relative price vector $\boldsymbol{z}_t$.
• Figure 2: Workflow of the proposed architecture.The state representation is first processed through dynamic clustering and masking, after which the upper-level agent extracts global signals and the lower-level agents perform intra-group allocation. A fusion mechanism then integrates the outputs to produce the final portfolio weights, ensuring that alongside concentrated investment in high-quality assets, the inclusion of remaining assets provides diversification benefits that reduce overall portfolio risk bib85.
• Figure 3: [Closed-loop interaction.]Closed-loop interaction between agent and environment in reinforcement learning, where the agent observes the state $s_t$, takes action $a_t$, and receives the reward $r_t$ together with the next state $s_{t+1}$.
• Figure 4: [Overall workflow of the proposed framework.]Overall workflow of the proposed hierarchical reinforcement learning framework bib15. The framework consists of an upper-level agent, two lower-level agents, and an environment module. The upper-level agent captures global market information and generates coordination signals, which guide the lower-level agents in asset-specific allocation. Each lower-level agent focuses on a distinct subset of assets, extracting local temporal patterns and making portfolio decisions accordingly. The environment provides market observations and reward feedback to complete the learning loop. Detailed descriptions of the module composition and implementation are available in the original work bib15.
• Figure 5: [Architecture of the lower-level actor network.]Architecture of the lower-level actor network. The network integrates masked logarithmic return matrices, global information from the upper-level agent, and a SLaK backbone to generate intra-group weight distributions, which are subsequently fused with prior signals to obtain group-wise portfolio weights.