Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

QuantEval: A Benchmark for Financial Quantitative Tasks in Large Language Models

Zhaolu Kang, Junhao Gong, Wenqing Hu, Shuo Yin, Kehan Jiang, Zhicheng Fang, Yingjie He, Chunlei Meng, Rong Fu, Dongyang Chen, Leqi Zheng, Eric Hanchen Jiang, Yunfei Feng, Yitong Leng, Junfan Zhu, Xiaoyou Chen, Xi Yang, Richeng Xuan

TL;DR

QuantEval introduces an execution-based benchmark for large language models in quantitative finance, spanning knowledge-based QA, multi-step quantitative reasoning, and strategy coding evaluated through a CTA-style backtesting framework. The authors construct 1,575 samples via expert annotation and multi-agent generation, and evaluate 13 models (open and closed) against human baselines, revealing substantial gaps, especially in reasoning and coding. They demonstrate that domain-aligned supervised fine-tuning and reinforcement learning yield consistent improvements, though the gap to human expertise remains sizable. By releasing deterministic backtesting configurations and evaluation scripts, QuantEval aims to accelerate research and practical deployment of LLMs in real-world trading workflows.

Abstract

Large Language Models (LLMs) have shown strong capabilities across many domains, yet their evaluation in financial quantitative tasks remains fragmented and mostly limited to knowledge-centric question answering. We introduce QuantEval, a benchmark that evaluates LLMs across three essential dimensions of quantitative finance: knowledge-based QA, quantitative mathematical reasoning, and quantitative strategy coding. Unlike prior financial benchmarks, QuantEval integrates a CTA-style backtesting framework that executes model-generated strategies and evaluates them using financial performance metrics, enabling a more realistic assessment of quantitative coding ability. We evaluate some state-of-the-art open-source and proprietary LLMs and observe substantial gaps to human experts, particularly in reasoning and strategy coding. Finally, we conduct large-scale supervised fine-tuning and reinforcement learning experiments on domain-aligned data, demonstrating consistent improvements. We hope QuantEval will facilitate research on LLMs' quantitative finance capabilities and accelerate their practical adoption in real-world trading workflows. We additionally release the full deterministic backtesting configuration (asset universe, cost model, and metric definitions) to ensure strict reproducibility.

QuantEval: A Benchmark for Financial Quantitative Tasks in Large Language Models

TL;DR

QuantEval introduces an execution-based benchmark for large language models in quantitative finance, spanning knowledge-based QA, multi-step quantitative reasoning, and strategy coding evaluated through a CTA-style backtesting framework. The authors construct 1,575 samples via expert annotation and multi-agent generation, and evaluate 13 models (open and closed) against human baselines, revealing substantial gaps, especially in reasoning and coding. They demonstrate that domain-aligned supervised fine-tuning and reinforcement learning yield consistent improvements, though the gap to human expertise remains sizable. By releasing deterministic backtesting configurations and evaluation scripts, QuantEval aims to accelerate research and practical deployment of LLMs in real-world trading workflows.

Abstract

Large Language Models (LLMs) have shown strong capabilities across many domains, yet their evaluation in financial quantitative tasks remains fragmented and mostly limited to knowledge-centric question answering. We introduce QuantEval, a benchmark that evaluates LLMs across three essential dimensions of quantitative finance: knowledge-based QA, quantitative mathematical reasoning, and quantitative strategy coding. Unlike prior financial benchmarks, QuantEval integrates a CTA-style backtesting framework that executes model-generated strategies and evaluates them using financial performance metrics, enabling a more realistic assessment of quantitative coding ability. We evaluate some state-of-the-art open-source and proprietary LLMs and observe substantial gaps to human experts, particularly in reasoning and strategy coding. Finally, we conduct large-scale supervised fine-tuning and reinforcement learning experiments on domain-aligned data, demonstrating consistent improvements. We hope QuantEval will facilitate research on LLMs' quantitative finance capabilities and accelerate their practical adoption in real-world trading workflows. We additionally release the full deterministic backtesting configuration (asset universe, cost model, and metric definitions) to ensure strict reproducibility.
Paper Structure (55 sections, 6 figures, 10 tables)

This paper contains 55 sections, 6 figures, 10 tables.

Table of Contents

  1. Introduction
  2. QuantEval
  3. Overview
  4. Data Preparation
  5. Data Construction
  6. Validation
  7. De-duplication, leakage prevention and difficulty control
  8. Implementation Details
  9. Data Statistics
  10. Evaluation on QuantEval
  11. Evaluation Setup
  12. Knowledge QA & Quantitative Reasoning.
  13. Strategy Coding (CoT-only).
  14. Human Baseline.
  15. Main Results on QuantEval
  16. ...and 40 more sections

Figures (6)

  • Figure 1: Examples from the three core tasks in QuantEval: knowledge-based question answering, quantitative mathematical reasoning, and quantitative strategy coding.
  • Figure 2: The QuantEval data construction pipeline consists of three main stages: Data Preparation, Data Construction, and Validation.
  • Figure 3: Performance curves of DianJin-R1-7B on test sets during reinforcement learning. Curves show steady improvement and stabilization, indicating effective reward-guided optimization.
  • Figure :
  • Figure :
  • ...and 1 more figures