Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning-Based Market Making as a Stochastic Control on Non-Stationary Limit Order Book Dynamics

Rafael Zimmer, Oswaldo Luiz do Valle Costa

TL;DR

The results suggest that the reinforcement learning agent can effectively be used under non-stationary market conditions, and that the proposed simulator-based environment can serve as a valuable tool for training and pre-training reinforcement learning agents in market-making scenarios.

Abstract

Reinforcement Learning has emerged as a promising framework for developing adaptive and data-driven strategies, enabling market makers to optimize decision-making policies based on interactions with the limit order book environment. This paper explores the integration of a reinforcement learning agent in a market-making context, where the underlying market dynamics have been explicitly modeled to capture observed stylized facts of real markets, including clustered order arrival times, non-stationary spreads and return drifts, stochastic order quantities and price volatility. These mechanisms aim to enhance stability of the resulting control agent, and serve to incorporate domain-specific knowledge into the agent policy learning process. Our contributions include a practical implementation of a market making agent based on the Proximal-Policy Optimization (PPO) algorithm, alongside a comparative evaluation of the agent's performance under varying market conditions via a simulator-based environment. As evidenced by our analysis of the financial return and risk metrics when compared to a closed-form optimal solution, our results suggest that the reinforcement learning agent can effectively be used under non-stationary market conditions, and that the proposed simulator-based environment can serve as a valuable tool for training and pre-training reinforcement learning agents in market-making scenarios.

Reinforcement Learning-Based Market Making as a Stochastic Control on Non-Stationary Limit Order Book Dynamics

TL;DR

The results suggest that the reinforcement learning agent can effectively be used under non-stationary market conditions, and that the proposed simulator-based environment can serve as a valuable tool for training and pre-training reinforcement learning agents in market-making scenarios.

Abstract

Reinforcement Learning has emerged as a promising framework for developing adaptive and data-driven strategies, enabling market makers to optimize decision-making policies based on interactions with the limit order book environment. This paper explores the integration of a reinforcement learning agent in a market-making context, where the underlying market dynamics have been explicitly modeled to capture observed stylized facts of real markets, including clustered order arrival times, non-stationary spreads and return drifts, stochastic order quantities and price volatility. These mechanisms aim to enhance stability of the resulting control agent, and serve to incorporate domain-specific knowledge into the agent policy learning process. Our contributions include a practical implementation of a market making agent based on the Proximal-Policy Optimization (PPO) algorithm, alongside a comparative evaluation of the agent's performance under varying market conditions via a simulator-based environment. As evidenced by our analysis of the financial return and risk metrics when compared to a closed-form optimal solution, our results suggest that the reinforcement learning agent can effectively be used under non-stationary market conditions, and that the proposed simulator-based environment can serve as a valuable tool for training and pre-training reinforcement learning agents in market-making scenarios.

Paper Structure

This paper contains 20 sections, 17 equations, 6 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Bibliography Review
  3. Data Type
  4. State, Action, and Reward Spaces
  5. Chosen Algorithms
  6. Methodology
  7. A Formal Description of the Environment
  8. Chosen State Space
  9. Chosen Action Space
  10. Episodic Reward Function and Returns
  11. State Transition Distribution
  12. Market Model Description and Environment Dynamics
  13. Decision Process
  14. Generalized Policy Iteration and Policy Gradient
  15. Benchmark Closed-Form Expression for Simplified Model
  16. ...and 5 more sections

Figures (6)

  • Figure 1: Data types used in the references.
  • Figure 2: Diagram of the Generalized Policy Iteration loop for the implemented agent.
  • Figure 3: Actor Network Architecture
  • Figure 4: Exponential moving average of the training reward per episode, with a linear trend line.
  • Figure 5: Financial return, averaged over 100 trajectories with a 1 standard deviation confidence interval.
  • ...and 1 more figures