Your Offline Policy is Not Trustworthy: Bilevel Reinforcement Learning for Sequential Portfolio Optimization

TL;DR

MetaTrader addresses the generalization-overfitting dilemma in offline RL for sequential portfolio optimization by formulating a partial-offline MDP with decoupled market and balance states, and by coupling data-transformations with a bilevel actor-critic learning loop. It introduces a transformation-based TD bootstrapping scheme that computes worst-case targets from a batch of transformed data, reducing value overestimation under distributional shifts. Across CSI-300 and NASDAQ-100, MetaTrader achieves superior results in offline and online settings, improving metrics such as cumulative return $CR$, annualized return $AR$, and Sharpe ratio $SR$ while reducing maximum drawdown $MDD$ relative to RL-based and predictor baselines. The work offers a robust and generalizable framework for non-stationary financial decision-making and lays a foundation for applying similar strategies to other dynamic domains like autonomous systems.

Abstract

Reinforcement learning (RL) has shown significant promise for sequential portfolio optimization tasks, such as stock trading, where the objective is to maximize cumulative returns while minimizing risks using historical data. However, traditional RL approaches often produce policies that merely memorize the optimal yet impractical buying and selling behaviors within the fixed dataset. These offline policies are less generalizable as they fail to account for the non-stationary nature of the market. Our approach, MetaTrader, frames portfolio optimization as a new type of partial-offline RL problem and makes two technical contributions. First, MetaTrader employs a bilevel learning framework that explicitly trains the RL agent to improve both in-domain profits on the original dataset and out-of-domain performance across diverse transformations of the raw financial data. Second, our approach incorporates a new temporal difference (TD) method that approximates worst-case TD estimates from a batch of transformed TD targets, addressing the value overestimation issue that is particularly challenging in scenarios with limited offline data. Our empirical results on two public stock datasets show that MetaTrader outperforms existing methods, including both RL-based approaches and traditional stock prediction models.

Figures (12)

• Figure 1: A comparison of MetaTrader and existing RL-based trading methods.a, Existing RL-for-finance methods typically adopt an offline training setup rather than online RL, causing them to struggle with the generalization-optimality dilemma, a common challenge in the inherently non-stationary financial market. b, MetaTrader tackles this paradox through: (1) specialized data transformations to simulate OOD financial data, (2) a bilevel RL framework that explicitly optimizes both in- and out-of-domain performance across diverse transformations, and (3) a novel TD learning method that conservatively estimates state-action values by approximating the minimum TD targets generated from a batch of data transformations.
• Figure 2: The MDP in the partial-offline RL setup for sequential portfolio optimization. The MDP consists of decoupled pairs of action-free market states and action-dependent balance states, with market states restricted to the offline training set. Unlike standard offline RL, where no new rewards are accessible during policy optimization, the partial-offline setup allows the agent to interact with the fixed training set, exploring different policies and collecting new reward feedback.
• Figure 3: The bilevel learning scheme of MetaTrader based on transformed market data. In the inner optimization loop (blue arrows), we optimize the model parameters on a batch of data subsets. In the outer optimization loop (red arrows), we perform bilevel gradient updates by explicitly evaluating the inner-loop parameters against another batch of data subsets. This process leads to a more generalizable agent and prevents overfitting to the in-domain optimal policy.
• Figure 4: An example of market data transformations.$F_1$ selects the top $\alpha\%$ of assets with the highest price gains and inverses their original growth rate to declines to simulate unexpected short-term disruptions. $F_2$ reverses the temporal order of a $T$-length sequence to simulate the long-term impact of certain events. $F_3$ downsamples the original data by $\Delta$ time steps to simulate squeezed global dynamics.
• Figure 5: Transformation-based TD learning with worst-case bootstrapping. The left part represents the TD estimate, and the right part corresponds to the TD target. By approximating worst-case future payoffs through a Monte Carlo method over a batch of data transformations, our approach aims to improve the generalizability of policies learned from offline data. This also helps mitigate the value overestimation issue, which is especially problematic when there are substantial discrepancies between the training and test distributions of non-stationary market data.