Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SysTradeBench: An Iterative Build-Test-Patch Benchmark for Strategy-to-Code Trading Systems with Drift-Aware Diagnostics

Yuchen Cao, Hanlin Zhang, Jacky Wai Keung, Yang Chen, Linqi Song

Abstract

Large language models (LLMs) are increasingly used as quantitative research copilots to translate natural-language strategy specifications into executable trading code. Yet most existing evaluations either focus on static financial knowledge or summarize performance with a single profitability metric, leaving a gap for benchmarking strategy-to-code trading systems as governed, auditable software. We introduce SysTradeBench (SysTB), an iterative build-test-patch benchmark that evaluates LLM-generated trading systems under drift-aware diagnostics. Given a standardized Base Strategy Doc and frozen semantics, each model must produce (i) a strategy card, (ii) executable code, and (iii) mandatory audit logs. A sandboxed harness runs determinism and anti-leakage checks, detects rule drift across iterations, and returns evidence bundles to support constrained patches. SysTradeBench reports multi-dimensional scorecards for spec fidelity, risk discipline, reliability, and out-of-sample robustness indicators, together with cost-effectiveness signals. We evaluate 17 models across 12 strategies. Top models achieve validity above 91.7 percent with strong aggregate scores, but evidence-driven iteration also induces code convergence by Iter2. These findings suggest that LLM iteration complements rather than replaces human quantitative researcher governance: LLMs excel at rapid prototyping and shallow bug fixes, while human oversight remains essential for critical strategies requiring solution diversity and ensemble robustness.

SysTradeBench: An Iterative Build-Test-Patch Benchmark for Strategy-to-Code Trading Systems with Drift-Aware Diagnostics

Abstract

Large language models (LLMs) are increasingly used as quantitative research copilots to translate natural-language strategy specifications into executable trading code. Yet most existing evaluations either focus on static financial knowledge or summarize performance with a single profitability metric, leaving a gap for benchmarking strategy-to-code trading systems as governed, auditable software. We introduce SysTradeBench (SysTB), an iterative build-test-patch benchmark that evaluates LLM-generated trading systems under drift-aware diagnostics. Given a standardized Base Strategy Doc and frozen semantics, each model must produce (i) a strategy card, (ii) executable code, and (iii) mandatory audit logs. A sandboxed harness runs determinism and anti-leakage checks, detects rule drift across iterations, and returns evidence bundles to support constrained patches. SysTradeBench reports multi-dimensional scorecards for spec fidelity, risk discipline, reliability, and out-of-sample robustness indicators, together with cost-effectiveness signals. We evaluate 17 models across 12 strategies. Top models achieve validity above 91.7 percent with strong aggregate scores, but evidence-driven iteration also induces code convergence by Iter2. These findings suggest that LLM iteration complements rather than replaces human quantitative researcher governance: LLMs excel at rapid prototyping and shallow bug fixes, while human oversight remains essential for critical strategies requiring solution diversity and ensemble robustness.

Paper Structure

This paper contains 62 sections, 17 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Financial-domain LLMs and data-centric adaptation
  4. Static finance benchmarks for QA and reasoning
  5. Code generation, program repair, and iterative evaluation
  6. Executable trading benchmarks and sequential decision making
  7. Backtest overfitting, frictions, and statistical validity
  8. Illustrative Example: When "Understanding" Diverges from Code Quality
  9. SysTradeBench: Benchmark Design
  10. System Architecture and Workflow
  11. Input Layer: Standardized Contracts and Data
  12. Strategy Specification Library
  13. Data Suite and Frozen Time Splits
  14. Evidence Bundles (Iter$k>0$)
  15. Model Generation Layer: Artifacts and Interfaces
  16. ...and 47 more sections

Figures (5)

  • Figure 1: System Architecture and Iterative Workflow of SysTradeBench
  • Figure 2: RQ3: Learning curves for Bollinger Mean Reversion strategy (3 top models × 4 iterations). (a) D1--D4 evolution. (b) OOS return trajectory showing Iter2 convergence trap (-4.6%) and Iter3 multi-objective recovery (+22.2%). (c) Average dimension scores (Iter1 vs Iter2). (d) Iteration gains showing diminishing returns (Iter1$\rightarrow$2: +0.08 avg).
  • Figure 3: RQ4: Token usage heatmap (3 models × 4 iterations). Request tokens stabilize after Iter0; response tokens shrink as models generate patches rather than full code.
  • Figure 4: Illustrative Example: Code Quality Issues Across Five LLMs
  • Figure 5: Iter0 cross-evaluation heatmap.