Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Right Place, Right Time: Market Simulation-based RL for Execution Optimisation

Ollie Olby, Andreea Bacalum, Rory Baggott, Namid Stillman

TL;DR

Addresses optimal execution of large meta-orders under realistic market dynamics by integrating a reinforcement learning framework with the Simudyne Pulse intraday market simulator to decompose slippage into market impact and execution risk. The RL agent learns distributions of order timings (including multi-modal schedules) and is evaluated against baselines (TWAP/VWAP) and the Almgren-Chriss efficient frontier, consistently approaching or beating baselines and aligning near the frontier. The study demonstrates that RL can uncover interpretable, risk-aware execution strategies, with performance that depends on the chosen objective (minimizing impact vs. minimizing slippage) and validates the approach using historical HK Hang Seng futures data for calibration. Overall, the work suggests RL is a viable, practical tool for discovering adaptive, frontier-leaning execution policies in high-fidelity market environments, with future work pointing toward contextual decision-making and frontier-based loss formulations.

Abstract

Execution algorithms are vital to modern trading, they enable market participants to execute large orders while minimising market impact and transaction costs. As these algorithms grow more sophisticated, optimising them becomes increasingly challenging. In this work, we present a reinforcement learning (RL) framework for discovering optimal execution strategies, evaluated within a reactive agent-based market simulator. This simulator creates reactive order flow and allows us to decompose slippage into its constituent components: market impact and execution risk. We assess the RL agent's performance using the efficient frontier based on work by Almgren and Chriss, measuring its ability to balance risk and cost. Results show that the RL-derived strategies consistently outperform baselines and operate near the efficient frontier, demonstrating a strong ability to optimise for risk and impact. These findings highlight the potential of reinforcement learning as a powerful tool in the trader's toolkit.

Right Place, Right Time: Market Simulation-based RL for Execution Optimisation

TL;DR

Addresses optimal execution of large meta-orders under realistic market dynamics by integrating a reinforcement learning framework with the Simudyne Pulse intraday market simulator to decompose slippage into market impact and execution risk. The RL agent learns distributions of order timings (including multi-modal schedules) and is evaluated against baselines (TWAP/VWAP) and the Almgren-Chriss efficient frontier, consistently approaching or beating baselines and aligning near the frontier. The study demonstrates that RL can uncover interpretable, risk-aware execution strategies, with performance that depends on the chosen objective (minimizing impact vs. minimizing slippage) and validates the approach using historical HK Hang Seng futures data for calibration. Overall, the work suggests RL is a viable, practical tool for discovering adaptive, frontier-leaning execution policies in high-fidelity market environments, with future work pointing toward contextual decision-making and frontier-based loss formulations.

Abstract

Execution algorithms are vital to modern trading, they enable market participants to execute large orders while minimising market impact and transaction costs. As these algorithms grow more sophisticated, optimising them becomes increasingly challenging. In this work, we present a reinforcement learning (RL) framework for discovering optimal execution strategies, evaluated within a reactive agent-based market simulator. This simulator creates reactive order flow and allows us to decompose slippage into its constituent components: market impact and execution risk. We assess the RL agent's performance using the efficient frontier based on work by Almgren and Chriss, measuring its ability to balance risk and cost. Results show that the RL-derived strategies consistently outperform baselines and operate near the efficient frontier, demonstrating a strong ability to optimise for risk and impact. These findings highlight the potential of reinforcement learning as a powerful tool in the trader's toolkit.
Paper Structure (16 sections, 10 equations, 4 figures, 2 tables)

This paper contains 16 sections, 10 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Agent-Based Simulators
  4. Optimal Trade Execution
  5. Methodology
  6. Market Simulator
  7. The Exchange
  8. The Agents
  9. Calibration
  10. Transaction Cost Calculation
  11. Reinforcement Learning
  12. Results and Discussion
  13. Distribution Modes
  14. RL on the Efficient Frontier
  15. Conclusion
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Overview of the proposed framework using the Simudyne Pulse intraday market simulator.
  • Figure 2: Price paths showing one MC run for the RL learned bi-modal un-bounded execution distribution and the two baselines (TWAP and VWAP). For the RL devised strategy the position and size of the trades are shown in black. The values for slippage are averaged over ten Monte Carlo runs. This is based on data from the Hang Seng futures index from 12th of September 2023
  • Figure 3: Plot showing the three different distributions found by the RL trading agent for each objective function. The plot also shows the aggregate volume over 1 minute periods in red. The lunch period is shown as a shaded region.
  • Figure 4: Plot showing the efficient frontier, defined by a comprehensive sample of TWAP strategies with time-differing start and end points. The three reference strategies are shown, where A is a risk-adverse strategy where all the available volume is executed as fast as possible, B is a risk-neutral strategy where the volume is executed evenly over the full day and C is a risk-seeking strategy where all the volume is executed at the end of the day. The dashed part of the frontier shows the sub-optimal part of the frontier, where similar transaction cost are incurred but with a higher amount of market risk. This region is defined as being to the right of the inflection point or where $\lambda =0$. The RL devised strategies are shown alongside the baselines. The tangent line at $\lambda = -1.645$ is shown, this shows the plane of strategies for an optimal VaR at a 95% confidence level.