Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FinForge: Semi-Synthetic Financial Benchmark Generation

Glenn Matlin, Akhil Theerthala, Anant Gupta, Anirudh JM, Rayan Castilla, Yi Mei Ng, Sudheer Chava

TL;DR

FinForge introduces a scalable, semi-synthetic pipeline to generate contamination-free, finance-domain benchmarks by fusing expert-guided data curation with controlled LM-based synthesis. The FinForge-5k dataset, drawn from a 143M-token corpus spanning 11 subdomains and 100k+ verified documents, enables robust evaluation of financial reasoning and multi-step problem solving. Benchmark results reveal that model scale alone does not guarantee strong financial reasoning, with open-source mid-range models sometimes competing with larger closed models, and highlight gaps in conceptual and quantitative finance capabilities. The framework demonstrates a practical, open-source approach for dynamic, domain-specific evaluation, with expert oversight playing a critical role in ensuring quality and relevance for real-world financial AI applications.

Abstract

Evaluating Language Models (LMs) in specialized, high-stakes domains such as finance remains a significant challenge due to the scarcity of open, high-quality, and domain-specific datasets. Existing general-purpose benchmarks provide broad coverage but lack the depth and domain fidelity needed to assess LMs' capabilities for real-world financial reasoning, which requires both conceptual understanding and quantitative rigor. To address this gap, we introduce FinForge, a scalable, semi-synthetic pipeline for constructing finance-specific evaluation benchmarks through a hybrid of expert-guided data curation and controlled LM-based synthesis. FinForge combines manual and programmatic corpus construction from authoritative financial sources with structured question generation and validation using Gemini 2.5 Flash. To demonstrate the pipeline's efficacy, we produce FinForge-5k, a snapshot benchmark comprising over 5,000 human-validated question-answer pairs across 11 finance subdomains, derived from a curated corpus of 100,000 verified documents totaling 143M tokens. Evaluation of state-of-the-art open-source and closed-source models on FinForge-5k reveals significant differences in financial reasoning, with leading models achieving accuracy levels near 80%. These findings underscore the framework's utility for diagnosing current model limitations and guiding future improvements in financial domain competence. All code and data are available at https://github.com/gtfintechlab/FinForge.

FinForge: Semi-Synthetic Financial Benchmark Generation

TL;DR

FinForge introduces a scalable, semi-synthetic pipeline to generate contamination-free, finance-domain benchmarks by fusing expert-guided data curation with controlled LM-based synthesis. The FinForge-5k dataset, drawn from a 143M-token corpus spanning 11 subdomains and 100k+ verified documents, enables robust evaluation of financial reasoning and multi-step problem solving. Benchmark results reveal that model scale alone does not guarantee strong financial reasoning, with open-source mid-range models sometimes competing with larger closed models, and highlight gaps in conceptual and quantitative finance capabilities. The framework demonstrates a practical, open-source approach for dynamic, domain-specific evaluation, with expert oversight playing a critical role in ensuring quality and relevance for real-world financial AI applications.

Abstract

Evaluating Language Models (LMs) in specialized, high-stakes domains such as finance remains a significant challenge due to the scarcity of open, high-quality, and domain-specific datasets. Existing general-purpose benchmarks provide broad coverage but lack the depth and domain fidelity needed to assess LMs' capabilities for real-world financial reasoning, which requires both conceptual understanding and quantitative rigor. To address this gap, we introduce FinForge, a scalable, semi-synthetic pipeline for constructing finance-specific evaluation benchmarks through a hybrid of expert-guided data curation and controlled LM-based synthesis. FinForge combines manual and programmatic corpus construction from authoritative financial sources with structured question generation and validation using Gemini 2.5 Flash. To demonstrate the pipeline's efficacy, we produce FinForge-5k, a snapshot benchmark comprising over 5,000 human-validated question-answer pairs across 11 finance subdomains, derived from a curated corpus of 100,000 verified documents totaling 143M tokens. Evaluation of state-of-the-art open-source and closed-source models on FinForge-5k reveals significant differences in financial reasoning, with leading models achieving accuracy levels near 80%. These findings underscore the framework's utility for diagnosing current model limitations and guiding future improvements in financial domain competence. All code and data are available at https://github.com/gtfintechlab/FinForge.
Paper Structure (16 sections, 1 figure, 1 table)

This paper contains 16 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Related Works
  3. General Knowledge and QA Benchmarks
  4. Financial QA Datasets and Benchmarks
  5. Methodology
  6. Data Curation
  7. Question Generation and Validation
  8. Synthetic Question Generation
  9. Validation and Filtering
  10. Results
  11. Model Benchmarking
  12. Expert Evaluation
  13. Addressing Data Contamination and Circularity
  14. Discussion
  15. Conclusion
  16. ...and 1 more sections

Figures (1)

  • Figure 1: The FinForge pipeline comprises two complementary stages. Data Curation (left): A hybrid manual--programmatic process that applies a financial taxonomy to identify authoritative web domains, scrapes and filters content, and assembles a high-quality Finance Corpus. Question Generation (right): An LM-driven five-stage workflow that analyzes documents to extract salient information, creates structured answer plans, generates self-contained questions with plausible distractors, assigns category labels, and applies rubric-based validation to ensure relevance, clarity, and factual accuracy. The validated outputs populate both a Q/A Corpus for benchmarking and feed back into the Finance Corpus for iterative refinement.