TraderBench: How Robust Are AI Agents in Adversarial Capital Markets?

Abstract

Evaluating AI agents in finance faces two key challenges: static benchmarks require costly expert annotation yet miss the dynamic decision-making central to real-world trading, while LLM-based judges introduce uncontrolled variance on domain-specific tasks. We introduce TraderBench, a benchmark that addresses both issues. It combines expert-verified static tasks (knowledge retrieval, analytical reasoning) with adversarial trading simulations scored purely on realized performance-Sharpe ratio, returns, and drawdown-eliminating judge variance entirely. The framework features two novel tracks: crypto trading with four progressive market-manipulation transforms, and options derivatives scoring across P&L accuracy, Greeks, and risk management. Trading scenarios can be refreshed with new market data to prevent benchmark contamination. Evaluating 13 models (8B open-source to frontier) on ~50 tasks, we find: (1) 8 of 13 models score ~33 on crypto with <1-point variation across adversarial conditions, exposing fixed non-adaptive strategies; (2) extended thinking helps retrieval (+26 points) but has zero impact on trading (+0.3 crypto, -0.1 options). These findings reveal that current agents lack genuine market adaptation, underscoring the need for performance-grounded evaluation in finance.

Figures (6)

• Figure 1: Overview of the TraderBench two-agent architecture. The Evaluator Agent generates tasks from six datasets and sends them to the Candidate Agent via the A2A protocol. The Candidate Agent uses an LLM and six MCP servers to access financial data, execute code, and simulate trades. Responses are scored by dataset-specific evaluators.
• Figure 2: Overall TraderBench scores for all 12 models, sorted by performance. The dashed line marks the 50/100 midpoint. Frontier proprietary models (left) clearly separate from smaller open-source models (right).
• Figure 3: Analytical Reasoning (self-contained) vs. Knowledge Retrieval (tool-dependent) scores. Models sorted by AR; hatched bars denote ablation variants (GPT-5.2+WS = web search, Qwen3-32B-T = thinking mode). Several base models score 75+ on reasoning but below 15 on retrieval. Both ablation variants show large KR gains, confirming that tool access and tool-use planning drive retrieval performance.
• Figure 4: Options trading sub-scores across all 12 models. P&L accuracy (80--93) consistently dominates Greeks precision (18--53), revealing a universal conceptual-vs-computational gap with a mean 54-point difference. Note the "competence mirage" where models correctly identify strategies but fail to quantify their risks.
• Figure 5: Crypto trading scores by adversarial transform condition. Seven models (bottom cluster, $\sim$32--34) show virtually no variation, suggesting a fixed non-adaptive strategy. Five models (top cluster, 45--52) actively trade; among these, only GPT-4o and Gemma3-27B show large transform-dependent variation.