Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning in Queue-Reactive Models: Application to Optimal Execution

Tomas Espana, Yadh Hafsi, Fabrizio Lillo, Edoardo Vittori

TL;DR

This work tackles the optimal execution problem for large meta-orders by minimizing execution costs in a microstructure realistic setting. It introduces a model free reinforcement learning approach using a Queue-Reactive Model to supply counterfactual feedback in a market that exhibits endogenous price impact and liquidity replenishment. A Double Deep Q-Network learns a policy that is both strategic and tactical, adapting to real time liquidity and price dynamics and outperforming standard baselines such as TWAP and fixed percentage of posted volume strategies. The proposed framework demonstrates that model free RL can yield robust, adaptive execution strategies and offers a path toward extensions to multi asset and signal integrated execution pipelines.

Abstract

We investigate the use of Reinforcement Learning for the optimal execution of meta-orders, where the objective is to execute incrementally large orders while minimizing implementation shortfall and market impact over an extended period of time. Departing from traditional parametric approaches to price dynamics and impact modeling, we adopt a model-free, data-driven framework. Since policy optimization requires counterfactual feedback that historical data cannot provide, we employ the Queue-Reactive Model to generate realistic and tractable limit order book simulations that encompass transient price impact, and nonlinear and dynamic order flow responses. Methodologically, we train a Double Deep Q-Network agent on a state space comprising time, inventory, price, and depth variables, and evaluate its performance against established benchmarks. Numerical simulation results show that the agent learns a policy that is both strategic and tactical, adapting effectively to order book conditions and outperforming standard approaches across multiple training configurations. These findings provide strong evidence that model-free Reinforcement Learning can yield adaptive and robust solutions to the optimal execution problem.

Reinforcement Learning in Queue-Reactive Models: Application to Optimal Execution

TL;DR

This work tackles the optimal execution problem for large meta-orders by minimizing execution costs in a microstructure realistic setting. It introduces a model free reinforcement learning approach using a Queue-Reactive Model to supply counterfactual feedback in a market that exhibits endogenous price impact and liquidity replenishment. A Double Deep Q-Network learns a policy that is both strategic and tactical, adapting to real time liquidity and price dynamics and outperforming standard baselines such as TWAP and fixed percentage of posted volume strategies. The proposed framework demonstrates that model free RL can yield robust, adaptive execution strategies and offers a path toward extensions to multi asset and signal integrated execution pipelines.

Abstract

We investigate the use of Reinforcement Learning for the optimal execution of meta-orders, where the objective is to execute incrementally large orders while minimizing implementation shortfall and market impact over an extended period of time. Departing from traditional parametric approaches to price dynamics and impact modeling, we adopt a model-free, data-driven framework. Since policy optimization requires counterfactual feedback that historical data cannot provide, we employ the Queue-Reactive Model to generate realistic and tractable limit order book simulations that encompass transient price impact, and nonlinear and dynamic order flow responses. Methodologically, we train a Double Deep Q-Network agent on a state space comprising time, inventory, price, and depth variables, and evaluate its performance against established benchmarks. Numerical simulation results show that the agent learns a policy that is both strategic and tactical, adapting effectively to order book conditions and outperforming standard approaches across multiple training configurations. These findings provide strong evidence that model-free Reinforcement Learning can yield adaptive and robust solutions to the optimal execution problem.

Paper Structure

This paper contains 37 sections, 1 theorem, 18 equations, 38 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works.
  3. Main Contributions.
  4. Problem Formulation
  5. The Queue-Reactive Model
  6. Description of the Market Simulation
  7. Invariant Distribution and Ergodicity
  8. Price Dynamics
  9. Market impact
  10. Optimal Execution Setting
  11. Markov Decision Process Embedding
  12. State space
  13. Action space
  14. Reward function
  15. Reinforcement Learning
  16. ...and 22 more sections

Key Result

Theorem 3.1

Assume that Then $X$ is non-explosive, irreducible, positive recurrent, and therefore ergodic with a unique invariant probability measure $\pi$.

Figures (38)

  • Figure 1: Proposed methodology: learning optimal execution strategies using RL.
  • Figure 2: QRM response to consuming the best ask $q_1$ at time $t$ with tick size $\delta=0.01$. The root node shows a typical LOB state just before the trade at time $t^-$, with volumes drawn from the invariant distribution, while the leaf nodes depict the possible post-trade configurations generated by the QRM dynamics at time $t$.
  • Figure 3: Average mid-price across $20{,}000$ simulations in which a trader systematically buys the entire best ask at fixed time intervals (vertical dashed lines). We set $\theta\!=\!0.7$.
  • Figure 4: Short term.
  • Figure 5: Long term.
  • ...and 33 more figures

Theorems & Definitions (4)

  • Remark 3.1
  • Theorem 3.1: Ergodicity
  • proof
  • Remark 3.2