Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PredictionMarketBench: A SWE-bench-Style Framework for Backtesting Trading Agents on Prediction Markets

Avi Arora, Ritesh Malpani

TL;DR

PredictionMarketBench presents a SWE-bench-style framework to backtest trading agents on prediction markets through deterministic, event-driven replay of real orderbooks, trades, and settlements. It introduces a standardized episode construction pipeline, an execution-realistic simulator with maker/taker semantics and fee modeling, and a tool-enabled agent interface for both classical strategies and LLM-based agents. The initial release provides four Kalshi-based episodes spanning cryptocurrency, weather, and sports, and shows that fee-aware algorithmic strategies can outperform naive approaches, while naive LLM behavior may incur sizable settlement losses. The benchmark enables apples-to-apples comparison, reproducibility, and systematic evaluation of agent designs under realistic execution constraints, with the goal of expanding datasets, baselines, and robust agent architectures in prediction-market trading research.

Abstract

Prediction markets offer a natural testbed for trading agents: contracts have binary payoffs, prices can be interpreted as probabilities, and realized performance depends critically on market microstructure, fees, and settlement risk. We introduce PredictionMarketBench, a SWE-bench-style benchmark for evaluating algorithmic and LLM-based trading agents on prediction markets via deterministic, event-driven replay of historical limit-order-book and trade data. PredictionMarketBench standardizes (i) episode construction from raw exchange streams (orderbooks, trades, lifecycle, settlement), (ii) an execution-realistic simulator with maker/taker semantics and fee modeling, and (iii) a tool-based agent interface that supports both classical strategies and tool-calling LLM agents with reproducible trajectories. We release four Kalshi-based episodes spanning cryptocurrency, weather, and sports. Baseline results show that naive trading agents can underperform due to transaction costs and settlement losses, while fee-aware algorithmic strategies remain competitive in volatile episodes.

PredictionMarketBench: A SWE-bench-Style Framework for Backtesting Trading Agents on Prediction Markets

TL;DR

PredictionMarketBench presents a SWE-bench-style framework to backtest trading agents on prediction markets through deterministic, event-driven replay of real orderbooks, trades, and settlements. It introduces a standardized episode construction pipeline, an execution-realistic simulator with maker/taker semantics and fee modeling, and a tool-enabled agent interface for both classical strategies and LLM-based agents. The initial release provides four Kalshi-based episodes spanning cryptocurrency, weather, and sports, and shows that fee-aware algorithmic strategies can outperform naive approaches, while naive LLM behavior may incur sizable settlement losses. The benchmark enables apples-to-apples comparison, reproducibility, and systematic evaluation of agent designs under realistic execution constraints, with the goal of expanding datasets, baselines, and robust agent architectures in prediction-market trading research.

Abstract

Prediction markets offer a natural testbed for trading agents: contracts have binary payoffs, prices can be interpreted as probabilities, and realized performance depends critically on market microstructure, fees, and settlement risk. We introduce PredictionMarketBench, a SWE-bench-style benchmark for evaluating algorithmic and LLM-based trading agents on prediction markets via deterministic, event-driven replay of historical limit-order-book and trade data. PredictionMarketBench standardizes (i) episode construction from raw exchange streams (orderbooks, trades, lifecycle, settlement), (ii) an execution-realistic simulator with maker/taker semantics and fee modeling, and (iii) a tool-based agent interface that supports both classical strategies and tool-calling LLM agents with reproducible trajectories. We release four Kalshi-based episodes spanning cryptocurrency, weather, and sports. Baseline results show that naive trading agents can underperform due to transaction costs and settlement losses, while fee-aware algorithmic strategies remain competitive in volatile episodes.
Paper Structure (37 sections, 1 figure, 6 tables)

This paper contains 37 sections, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. PredictionMarketBench
  3. Benchmark Construction
  4. Data collection.
  5. Episode representation.
  6. Harness and simulator.
  7. Task formulation
  8. Decision points and termination.
  9. Observations.
  10. Actions.
  11. Features of PredictionMarketBench
  12. Binary contract semantics.
  13. Maker/taker execution realism.
  14. Fee-aware evaluation.
  15. Deterministic, event-driven replay.
  16. ...and 22 more sections

Figures (1)

  • Figure 1: PredictionMarketBench execution flow. The harness iterates over episodes, the simulator replays historical events, and the agent interacts through a Kalshi-like context API at a fixed cadence.