Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Reinforcement Learning in Finance: Modeling Market Impact with Elliptic Uncertainty Sets

Shaocong Ma, Heng Huang

TL;DR

This work addresses the mis-match between training and deployment in financial RL by explicitly modeling market impact with elliptic uncertainty sets, which capture directional perturbations in prices. It develops both implicit and explicit closed-form solutions for the worst-case perturbation $u^*$ and integrates them into a robust TD-learning framework, enabling efficient policy evaluation under market impact. Empirical results on single-asset intra-day trading and multi-asset portfolio rebalancing show that the elliptic-uncertainty approach yields superior risk-adjusted returns (Sharpe ratio) and robustness to increasing trading volumes compared to non-robust RL and symmetric-ball baselines. The contributions offer a scalable, more faithful representation of market impact for robust RL, with practical implications for deploying stable, high-performance trading agents.

Abstract

In financial applications, reinforcement learning (RL) agents are commonly trained on historical data, where their actions do not influence prices. However, during deployment, these agents trade in live markets where their own transactions can shift asset prices, a phenomenon known as market impact. This mismatch between training and deployment environments can significantly degrade performance. Traditional robust RL approaches address this model misspecification by optimizing the worst-case performance over a set of uncertainties, but typically rely on symmetric structures that fail to capture the directional nature of market impact. To address this issue, we develop a novel class of elliptic uncertainty sets. We establish both implicit and explicit closed-form solutions for the worst-case uncertainty under these sets, enabling efficient and tractable robust policy evaluation. Experiments on single-asset and multi-asset trading tasks demonstrate that our method achieves superior Sharpe ratio and remains robust under increasing trade volumes, offering a more faithful and scalable approach to RL in financial markets.

Robust Reinforcement Learning in Finance: Modeling Market Impact with Elliptic Uncertainty Sets

TL;DR

This work addresses the mis-match between training and deployment in financial RL by explicitly modeling market impact with elliptic uncertainty sets, which capture directional perturbations in prices. It develops both implicit and explicit closed-form solutions for the worst-case perturbation and integrates them into a robust TD-learning framework, enabling efficient policy evaluation under market impact. Empirical results on single-asset intra-day trading and multi-asset portfolio rebalancing show that the elliptic-uncertainty approach yields superior risk-adjusted returns (Sharpe ratio) and robustness to increasing trading volumes compared to non-robust RL and symmetric-ball baselines. The contributions offer a scalable, more faithful representation of market impact for robust RL, with practical implications for deploying stable, high-performance trading agents.

Abstract

In financial applications, reinforcement learning (RL) agents are commonly trained on historical data, where their actions do not influence prices. However, during deployment, these agents trade in live markets where their own transactions can shift asset prices, a phenomenon known as market impact. This mismatch between training and deployment environments can significantly degrade performance. Traditional robust RL approaches address this model misspecification by optimizing the worst-case performance over a set of uncertainties, but typically rely on symmetric structures that fail to capture the directional nature of market impact. To address this issue, we develop a novel class of elliptic uncertainty sets. We establish both implicit and explicit closed-form solutions for the worst-case uncertainty under these sets, enabling efficient and tractable robust policy evaluation. Experiments on single-asset and multi-asset trading tasks demonstrate that our method achieves superior Sharpe ratio and remains robust under increasing trade volumes, offering a more faithful and scalable approach to RL in financial markets.
Paper Structure (59 sections, 15 theorems, 120 equations, 5 figures, 4 tables, 1 algorithm)

This paper contains 59 sections, 15 theorems, 120 equations, 5 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Existing Approaches to Handle Market Impact
  4. Robust RL in Finance
  5. Modeling the Uncertainty Set in Robust RL
  6. Preliminaries: Robust Reinforcement Learning
  7. Modeling Market Impact with Elliptic Uncertainty Sets
  8. Limitations of Symmetric Uncertainty Sets
  9. On the Conservativeness of Robust Policies
  10. The Elliptic Uncertainty Sets
  11. Connection to the Classical Ellipse
  12. Limitations & Potential Directions
  13. Solving the Worst-Case Uncertainty
  14. Robust TD Learning Algorithm
  15. Experiments
  16. ...and 44 more sections

Key Result

Theorem 3.4

Let $d:= |\mathcal{S}|$ be the cardinal of state space. Suppose that $\{u_n\}_{n=1}^N \subset \mathbb{R}^d$ for each $(s,a)\in \mathcal{S} \times \mathcal{A}$, the uncertainty size $\beta\geq 0$, and $v\in\mathbb{R}^d$. Then there exists $\lambda^*$ and $\mu^*$ such that the optimization problem def

Figures (5)

  • Figure 1: Market impact illustration using AMZN stock on June 21, 2012, based on 5-level LOBSTER data huang2011lobster. The left panel shows the price response to executing a buy order of $100$ shares at the time $t = 0\ \text{ms}$, which consumes ask-side liquidity within $1000\ \text{ms}$ and induces an immediate upward shift in price. The right panel shows the analogous impact of a sell order. This plot indicates that the transition dynamics induced by trading are not symmetrically distributed around the nominal kernel.
  • Figure 2: Illustration of the transition kernel in a simplified robust RL setting using an $\ell_\infty$-norm uncertainty set. Only one of the upward (buy) or downward (sell) shift is plausible in a given scenario; however, the ball-shaped uncertainty set must include both shifts due to its symmetric structure, which motivates us to propose an uncertainty set with non-symmetric structures.
  • Figure 3: Performance comparison of trading strategies on the META stock from June 9 to December 9, 2022, under simulated market impact. The left panel compares the final portfolio values with (in red) and without market impact (in green), illustrating the robustness of each method to execution-related slippage. The right panel shows the cumulative returns over the evaluation period, tracking the performance of the four strategies in \ref{['tab:performance']}, alongside the baseline performance of the META stock.
  • Figure 4: Robustness of RL agents to market impact under increasing trading volumes. The left three panels show normalized portfolio values over time across initial cash levels, with dashed and solid lines indicating performance with and without market impact, respectively. The right panel shows the relative portfolio gap, which increases sharply for the non-robust agent but remains small and stable for the robust RL agent with $\ell_p$-elliptic uncertainty sets.
  • Figure 5: Performance comparison of trading strategies on the MSFT stock and SPY ETF.

Theorems & Definitions (38)

  • Example 3.1: Symmetric Sets Fail to Capture Directional Uncertainty
  • Definition 3.2: Elliptic $(s,a)$-Uncertainty Set
  • Theorem 3.4: Implicit
  • Remark
  • Theorem 3.5: Explicit
  • Remark
  • Remark
  • Definition A.1: Signed Permutation Group
  • Proposition A.2
  • proof
  • ...and 28 more