Reinforcement Learning with Maskable Stock Representation for Portfolio Management in Customizable Stock Pools

TL;DR

This paper tackles portfolio management under customizable stock pools by introducing EarnMore, a one-shot RL framework trained on a global stock pool. It leverages a maskable stock representation with a two-level MAE-like masking and reconstruction to unify CSPs of different sizes, and couples this with a re-weighted SAC objective to concentrate investments on favorable stocks. Key contributions include a maskable token mechanism, a pool-level self-supervised embedding, and a temperature-based sparsification strategy that together yield superior ARR and SR across multiple CSPs and baselines. The approach offers practical promise for investor-specific, dynamically adjustable portfolios with scalable training and robust performance in diverse market conditions.

Abstract

Portfolio management (PM) is a fundamental financial trading task, which explores the optimal periodical reallocation of capitals into different stocks to pursue long-term profits. Reinforcement learning (RL) has recently shown its potential to train profitable agents for PM through interacting with financial markets. However, existing work mostly focuses on fixed stock pools, which is inconsistent with investors' practical demand. Specifically, the target stock pool of different investors varies dramatically due to their discrepancy on market states and individual investors may temporally adjust stocks they desire to trade (e.g., adding one popular stocks), which lead to customizable stock pools (CSPs). Existing RL methods require to retrain RL agents even with a tiny change of the stock pool, which leads to high computational cost and unstable performance. To tackle this challenge, we propose EarnMore, a rEinforcement leARNing framework with Maskable stOck REpresentation to handle PM with CSPs through one-shot training in a global stock pool (GSP). Specifically, we first introduce a mechanism to mask out the representation of the stocks outside the target pool. Second, we learn meaningful stock representations through a self-supervised masking and reconstruction process. Third, a re-weighting mechanism is designed to make the portfolio concentrate on favorable stocks and neglect the stocks outside the target pool. Through extensive experiments on 8 subset stock pools of the US stock market, we demonstrate that EarnMore significantly outperforms 14 state-of-the-art baselines in terms of 6 popular financial metrics with over 40% improvement on profit.

Figures (6)

• Figure 1: Overview of portfolio management by EarnMore in customizable stock pools (CSPs).
• Figure 2: The overall architecture of EarnMore. Module (a) is used to extract stock-level embeddings from GSP. Module (b) is the masking and reconstruction process to learn pool-level embeddings. Module (c) is an agent with masked token awareness.
• Figure 3: Performance on GSP for SP500 and DJ30
• Figure 4: Performance on CSPs with dynamic changes
• Figure 5: (a) Comparing the performance of EarnMore with direct methods on DJ30. (b) Comparing the time costs of EarnMore with several other methods on DJ30.

Theorems & Definitions (4)

• Definition 1
• Definition 2
• Definition 3
• Definition 4