AlphaForgeBench: Benchmarking End-to-End Trading Strategy Design with Large Language Models

TL;DR

Experiments across multiple state-of-the-art LLMs show that AlphaForgeBench eliminates execution-induced instability and provides a rigorous benchmark for assessing financial reasoning, strategy formulation, and alpha discovery.

Abstract

The rapid advancement of Large Language Models (LLMs) has led to a surge of financial benchmarks, evolving from static knowledge tests to interactive trading simulations. However, current evaluations of real-time trading performance overlook a critical failure mode: severe behavioral instability in sequential decision-making under uncertainty. We empirically show that LLM-based trading agents exhibit extreme run-to-run variance, inconsistent action sequences even under deterministic decoding, and irrational action flipping across adjacent time steps. These issues stem from stateless autoregressive architectures lacking persistent action memory, as well as sensitivity to continuous-to-discrete action mappings in portfolio allocation. As a result, many existing financial trading benchmarks produce unreliable, non-reproducible, and uninformative evaluations. To address these limitations, we propose AlphaForgeBench, a principled framework that reframes LLMs as quantitative researchers rather than execution agents. Instead of emitting trading actions, LLMs generate executable alpha factors and factor-based strategies grounded in financial reasoning. This design decouples reasoning from execution, enabling fully deterministic and reproducible evaluation while aligning with real-world quantitative research workflows. Experiments across multiple state-of-the-art LLMs show that AlphaForgeBench eliminates execution-induced instability and provides a rigorous benchmark for assessing financial reasoning, strategy formulation, and alpha discovery.

Figures (51)

• Figure 1: The framework of AlphaForgeBench.
• Figure 2: Multi-metric radar profiles on LLM-augmented queries (five metrics, $\tau{=}0$ vs. $\tau{=}0.7$). Near-identical polygons confirm temperature-invariant model behavior.
• Figure 3: Sharpe Ratio by difficulty level on LLM-augmented queries ($\tau{=}0$ vs. $\tau{=}0.7$). Inter-model spread widens monotonically from Level 1 to Level 3.
• Figure 4: Sharpe Ratio by asset on LLM-augmented queries ($\tau{=}0$ vs. $\tau{=}0.7$). A shared difficulty gradient emerges, with AAPL/GOOGL easiest and crypto/NVDA hardest.
• Figure 5: Aligned cumulative return curves on LLM-augmented queries ($\tau{=}0$ vs. $\tau{=}0.7$). Shaded bands denote the 25th--75th percentile range over 5 runs.