Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Realistic Market Impact Modeling for Reinforcement Learning Trading Environments

Lucas Riera Abbade, Anna Helena Reali Costa

Abstract

Reinforcement learning (RL) has shown promise for trading, yet most open-source backtesting environments assume negligible or fixed transaction costs, causing agents to learn trading behaviors that fail under realistic execution. We introduce three Gymnasium-compatible trading environments -- MACE (Market-Adjusted Cost Execution) stock trading, margin trading, and portfolio optimization -- that integrate nonlinear market impact models grounded in the Almgren-Chriss framework and the empirically validated square-root impact law. Each environment provides pluggable cost models, permanent impact tracking with exponential decay, and comprehensive trade-level logging. We evaluate five DRL algorithms (A2C, PPO, DDPG, SAC, TD3) on the NASDAQ-100, comparing a fixed 10 bps baseline against the AC model with Optuna-tuned hyperparameters. Our results show that (i) the cost model materially changes both absolute performance and the relative ranking of algorithms across all three environments; (ii) the AC model produces dramatically different trading behavior, e.g., daily costs dropping from $200k to $8k with turnover falling from 19% to 1%; (iii) hyperparameter optimization is essential for constraining pathological trading, with costs dropping up to 82%; and (iv) algorithm-cost model interactions are strongly environment-specific, e.g., DDPG's OOS Sharpe jumps from -2.1 to 0.3 under AC in margin trading while SAC's drops from -0.5 to -1.2. We release the full suite as an open-source extension to FinRL-Meta.

Realistic Market Impact Modeling for Reinforcement Learning Trading Environments

Abstract

Reinforcement learning (RL) has shown promise for trading, yet most open-source backtesting environments assume negligible or fixed transaction costs, causing agents to learn trading behaviors that fail under realistic execution. We introduce three Gymnasium-compatible trading environments -- MACE (Market-Adjusted Cost Execution) stock trading, margin trading, and portfolio optimization -- that integrate nonlinear market impact models grounded in the Almgren-Chriss framework and the empirically validated square-root impact law. Each environment provides pluggable cost models, permanent impact tracking with exponential decay, and comprehensive trade-level logging. We evaluate five DRL algorithms (A2C, PPO, DDPG, SAC, TD3) on the NASDAQ-100, comparing a fixed 10 bps baseline against the AC model with Optuna-tuned hyperparameters. Our results show that (i) the cost model materially changes both absolute performance and the relative ranking of algorithms across all three environments; (ii) the AC model produces dramatically different trading behavior, e.g., daily costs dropping from 8k with turnover falling from 19% to 1%; (iii) hyperparameter optimization is essential for constraining pathological trading, with costs dropping up to 82%; and (iv) algorithm-cost model interactions are strongly environment-specific, e.g., DDPG's OOS Sharpe jumps from -2.1 to 0.3 under AC in margin trading while SAC's drops from -0.5 to -1.2. We release the full suite as an open-source extension to FinRL-Meta.

Paper Structure

This paper contains 22 sections, 4 equations, 8 figures.

Table of Contents

  1. Introduction
  2. Market Impact Models
  3. The Square-Root Impact Law
  4. Almgren--Chriss Cost Decomposition
  5. Permanent Impact Decay
  6. Environment Design
  7. State Space
  8. Action Space and Trade Execution
  9. Stock and Margin Trading
  10. Portfolio Optimization
  11. Reward Function
  12. Experimental Setup
  13. Data and Universe
  14. DRL Algorithms
  15. Hyperparameter Optimization
  16. ...and 7 more sections

Figures (8)

  • Figure 1: OOS return---MACE stock trading, all five agents under baseline vs. AC impact, optimized params. Black line: QQEW benchmark (19%).
  • Figure 2: Non-optimized TD3 trading costs---MACE stock trading, baseline vs. AC cost model.
  • Figure 3: Average order POV per epoch---MACE stock trading, PPO, optimized vs. non-optimized (AC impact model).
  • Figure 4: OOS portfolio value---margin trading, A2C/PPO/DDPG/SAC, baseline vs. AC impact model.
  • Figure 5: PPO average order POV per epoch---margin trading.
  • ...and 3 more figures