Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LOBERT: Generative AI Foundation Model for Limit Order Book Messages

Eljas Linna, Kestutis Baltakys, Alexandros Iosifidis, Juho Kanniainen

TL;DR

LOBERT reframes limit order book message dynamics as a language modeling problem by introducing a one-token-per-message encoder with Piecewise Linear–Geometric Scaling and continuous-time rotary attention. The model integrates discrete message tokens with continuous price, volume, and time information, and employs a two-phase training regime of masked message modeling followed by task-specific fine-tuning, with a hybrid token–regressor decoding head. Empirical results on Nasdaq ITCH data show improved next-message and mid-price prediction performance, better distributional fidelity than baselines, and favorable behavior under confidence-based selective prediction, albeit with slower inference than some baselines. This approach demonstrates the potential of NLP-style foundation models for LOB tasks and opens avenues for scalable fine-tuning to diverse trading analytics and simulations, while highlighting areas for efficiency and long-horizon sequence validation.

Abstract

Modeling the dynamics of financial Limit Order Books (LOB) at the message level is challenging due to irregular event timing, rapid regime shifts, and the reactions of high-frequency traders to visible order flow. Previous LOB models require cumbersome data representations and lack adaptability outside their original tasks, leading us to introduce LOBERT, a general-purpose encoder-only foundation model for LOB data suitable for downstream fine-tuning. LOBERT adapts the original BERT architecture for LOB data by using a novel tokenization scheme that treats complete multi-dimensional messages as single tokens while retaining continuous representations of price, volume, and time. With these methods, LOBERT achieves leading performance in tasks such as predicting mid-price movements and next messages, while reducing the required context length compared to previous methods.

LOBERT: Generative AI Foundation Model for Limit Order Book Messages

TL;DR

LOBERT reframes limit order book message dynamics as a language modeling problem by introducing a one-token-per-message encoder with Piecewise Linear–Geometric Scaling and continuous-time rotary attention. The model integrates discrete message tokens with continuous price, volume, and time information, and employs a two-phase training regime of masked message modeling followed by task-specific fine-tuning, with a hybrid token–regressor decoding head. Empirical results on Nasdaq ITCH data show improved next-message and mid-price prediction performance, better distributional fidelity than baselines, and favorable behavior under confidence-based selective prediction, albeit with slower inference than some baselines. This approach demonstrates the potential of NLP-style foundation models for LOB tasks and opens avenues for scalable fine-tuning to diverse trading analytics and simulations, while highlighting areas for efficiency and long-horizon sequence validation.

Abstract

Modeling the dynamics of financial Limit Order Books (LOB) at the message level is challenging due to irregular event timing, rapid regime shifts, and the reactions of high-frequency traders to visible order flow. Previous LOB models require cumbersome data representations and lack adaptability outside their original tasks, leading us to introduce LOBERT, a general-purpose encoder-only foundation model for LOB data suitable for downstream fine-tuning. LOBERT adapts the original BERT architecture for LOB data by using a novel tokenization scheme that treats complete multi-dimensional messages as single tokens while retaining continuous representations of price, volume, and time. With these methods, LOBERT achieves leading performance in tasks such as predicting mid-price movements and next messages, while reducing the required context length compared to previous methods.

Paper Structure

This paper contains 17 sections, 9 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Message preprocessing
  4. Order Book snapshot preprocessing
  5. Model architecture, inputs and outputs
  6. Training and fine-tuning
  7. Experiments and results
  8. Next-message prediction experiment
  9. Message processing comparison
  10. Mid-price prediction experiment
  11. Discussion and conclusion
  12. Related work
  13. Piecewise Linear-Geometric Scaling
  14. Snapshot scaling
  15. Inference speed
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Simplified architecture diagram highlighting the unique properties of LOBERT
  • Figure 2: Value distributions of price, volume, time difference, type and side for LOBERT model with Book Module enabled.
  • Figure 3: Comparison of LOBERT and DeepLOB for mid-price prediction plotted by confidence threshold, F1 macro score and coverage for MSFT and AAPL assets.
  • Figure 4: Comparison of LOBERT and DeepLOB for mid-price prediction plotted by confidence threshold, F1 macro score and coverage for FB and INTC assets.