Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MacroHFT: Memory Augmented Context-aware Reinforcement Learning On High Frequency Trading

Chuqiao Zong, Chaojie Wang, Molei Qin, Lei Feng, Xinrun Wang, Bo An

TL;DR

MacroHFT is a novel Memory Augmented Context-aware Reinforcement learning method for HFT, equipped with a memory mechanism to enhance the capability of decision-making, which can achieve state-of-the-art performance on minute-level trading tasks.

Abstract

High-frequency trading (HFT) that executes algorithmic trading in short time scales, has recently occupied the majority of cryptocurrency market. Besides traditional quantitative trading methods, reinforcement learning (RL) has become another appealing approach for HFT due to its terrific ability of handling high-dimensional financial data and solving sophisticated sequential decision-making problems, \emph{e.g.,} hierarchical reinforcement learning (HRL) has shown its promising performance on second-level HFT by training a router to select only one sub-agent from the agent pool to execute the current transaction. However, existing RL methods for HFT still have some defects: 1) standard RL-based trading agents suffer from the overfitting issue, preventing them from making effective policy adjustments based on financial context; 2) due to the rapid changes in market conditions, investment decisions made by an individual agent are usually one-sided and highly biased, which might lead to significant loss in extreme markets. To tackle these problems, we propose a novel Memory Augmented Context-aware Reinforcement learning method On HFT, \emph{a.k.a.} MacroHFT, which consists of two training phases: 1) we first train multiple types of sub-agents with the market data decomposed according to various financial indicators, specifically market trend and volatility, where each agent owns a conditional adapter to adjust its trading policy according to market conditions; 2) then we train a hyper-agent to mix the decisions from these sub-agents and output a consistently profitable meta-policy to handle rapid market fluctuations, equipped with a memory mechanism to enhance the capability of decision-making. Extensive experiments on various cryptocurrency markets demonstrate that MacroHFT can achieve state-of-the-art performance on minute-level trading tasks.

MacroHFT: Memory Augmented Context-aware Reinforcement Learning On High Frequency Trading

TL;DR

MacroHFT is a novel Memory Augmented Context-aware Reinforcement learning method for HFT, equipped with a memory mechanism to enhance the capability of decision-making, which can achieve state-of-the-art performance on minute-level trading tasks.

Abstract

High-frequency trading (HFT) that executes algorithmic trading in short time scales, has recently occupied the majority of cryptocurrency market. Besides traditional quantitative trading methods, reinforcement learning (RL) has become another appealing approach for HFT due to its terrific ability of handling high-dimensional financial data and solving sophisticated sequential decision-making problems, \emph{e.g.,} hierarchical reinforcement learning (HRL) has shown its promising performance on second-level HFT by training a router to select only one sub-agent from the agent pool to execute the current transaction. However, existing RL methods for HFT still have some defects: 1) standard RL-based trading agents suffer from the overfitting issue, preventing them from making effective policy adjustments based on financial context; 2) due to the rapid changes in market conditions, investment decisions made by an individual agent are usually one-sided and highly biased, which might lead to significant loss in extreme markets. To tackle these problems, we propose a novel Memory Augmented Context-aware Reinforcement learning method On HFT, \emph{a.k.a.} MacroHFT, which consists of two training phases: 1) we first train multiple types of sub-agents with the market data decomposed according to various financial indicators, specifically market trend and volatility, where each agent owns a conditional adapter to adjust its trading policy according to market conditions; 2) then we train a hyper-agent to mix the decisions from these sub-agents and output a consistently profitable meta-policy to handle rapid market fluctuations, equipped with a memory mechanism to enhance the capability of decision-making. Extensive experiments on various cryptocurrency markets demonstrate that MacroHFT can achieve state-of-the-art performance on minute-level trading tasks.
Paper Structure (21 sections, 10 equations, 6 figures, 4 tables)

This paper contains 21 sections, 10 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Traditional Financial Methods
  4. RL-based Methods
  5. PRELIMINARIES
  6. Financial Definitions
  7. MDP Formulation
  8. MacroHFT
  9. Market Decomposition
  10. Low-Level Policy Optimization with Conditional Adaptation
  11. Meta-Policy Optimization with Memory Augmentation
  12. Experiments
  13. Datasets
  14. Evaluation Metrics
  15. Baselines
  16. ...and 6 more sections

Figures (6)

  • Figure 1: A Snapshot of Limit Order Book (LOB)
  • Figure 2: The overview of MacroHFT. In phase I, we train multiple types of sub-agents with conditional adapters on the market data decomposed according to trend and volatility indicators. In phase II, we train a hyper-agent to mix decisions from all sub-agents, enhanced with a memory mechanism.
  • Figure 3: Performance of MacroHFT and other baselines
  • Figure 4: Trading examples of different cryptocurrencies
  • Figure 5: Weight of sub-agents assigned by hyper-agent in BTCUSDT
  • ...and 1 more figures