Reinforcement Learning in Agent-Based Market Simulation: Unveiling Realistic Stylized Facts and Behavior
Zhiyuan Yao, Zheng Li, Matthew Thomas, Ionut Florescu
TL;DR
The paper tackles the need for realistic market simulators that reproduce stylized facts and responsiveness to external shocks. It introduces an agent-based continuous double auction market in which two RL agent classes—Market-Making (MM) and Liquidity Taking (LT)—learn via Proximal Policy Optimization, each with independent policies and heterogeneous hyperparameters. Empirical results show that the RL-based market reproduces heavy-tailed return distributions, absence of return autocorrelations, long-range volatility clustering, and plausible price impact under flash events, outperforming zero‑intelligence baselines. The study also demonstrates that continual learning during simulation yields the most realistic, adaptable behavior, highlighting practical relevance for regulators and investors.
Abstract
Investors and regulators can greatly benefit from a realistic market simulator that enables them to anticipate the consequences of their decisions in real markets. However, traditional rule-based market simulators often fall short in accurately capturing the dynamic behavior of market participants, particularly in response to external market impact events or changes in the behavior of other participants. In this study, we explore an agent-based simulation framework employing reinforcement learning (RL) agents. We present the implementation details of these RL agents and demonstrate that the simulated market exhibits realistic stylized facts observed in real-world markets. Furthermore, we investigate the behavior of RL agents when confronted with external market impacts, such as a flash crash. Our findings shed light on the effectiveness and adaptability of RL-based agents within the simulation, offering insights into their response to significant market events.