Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FinanceMath: Knowledge-Intensive Math Reasoning in Finance Domains

Yilun Zhao, Hongjun Liu, Yitao Long, Rui Zhang, Chen Zhao, Arman Cohan

TL;DR

FinanceMath introduces a knowledge-intensive finance math reasoning benchmark with 1,200 mixed-text and tabular problems and expert-annotated Python solutions. It pairs a finance knowledge bank of 864 terms with a 200/1,000 development/test split and evaluates 51 models from 16 organizations using Chain-of-Thought and Program-of-Thought prompting. Results show a large gap to human expert performance (GPT-4o at $60.9\%$ vs $92\%$ open-book) and demonstrate that knowledge augmentation via retrieval and term-definition strategies yields consistent gains, though still short of expert performance. The work provides insights into domain-specific knowledge retrieval and sets a foundation for improved reasoning in finance through targeted knowledge integration and executable-program reasoning.

Abstract

We introduce FinanceMath, a novel benchmark designed to evaluate LLMs' capabilities in solving knowledge-intensive math reasoning problems. Compared to prior works, this study features three core advancements. First, FinanceMath includes 1,200 problems with a hybrid of textual and tabular content. These problems require college-level knowledge in the finance domain for effective resolution. Second, we provide expert-annotated, detailed solution references in Python program format, ensuring a high-quality benchmark for LLM assessment. We also construct a finance-domain knowledge bank and investigate various knowledge integration strategies. Finally, we evaluate a wide spectrum of 44 LLMs with both Chain-of-Thought and Program-of-Thought prompting methods. Our experimental results reveal that the current best-performing system (i.e., GPT-4o) achieves only 60.9% accuracy using CoT prompting, leaving substantial room for improvement. Moreover, while augmenting LLMs with external knowledge can improve model performance (e.g., from 47.5% to 54.5% for Gemini-1.5-Pro), their accuracy remains significantly lower than the estimated human expert performance of 92%. We believe that FinanceMath can advance future research in the area of domain-specific knowledge retrieval and integration, particularly within the context of solving reasoning-intensive tasks.

FinanceMath: Knowledge-Intensive Math Reasoning in Finance Domains

TL;DR

FinanceMath introduces a knowledge-intensive finance math reasoning benchmark with 1,200 mixed-text and tabular problems and expert-annotated Python solutions. It pairs a finance knowledge bank of 864 terms with a 200/1,000 development/test split and evaluates 51 models from 16 organizations using Chain-of-Thought and Program-of-Thought prompting. Results show a large gap to human expert performance (GPT-4o at vs open-book) and demonstrate that knowledge augmentation via retrieval and term-definition strategies yields consistent gains, though still short of expert performance. The work provides insights into domain-specific knowledge retrieval and sets a foundation for improved reasoning in finance through targeted knowledge integration and executable-program reasoning.

Abstract

We introduce FinanceMath, a novel benchmark designed to evaluate LLMs' capabilities in solving knowledge-intensive math reasoning problems. Compared to prior works, this study features three core advancements. First, FinanceMath includes 1,200 problems with a hybrid of textual and tabular content. These problems require college-level knowledge in the finance domain for effective resolution. Second, we provide expert-annotated, detailed solution references in Python program format, ensuring a high-quality benchmark for LLM assessment. We also construct a finance-domain knowledge bank and investigate various knowledge integration strategies. Finally, we evaluate a wide spectrum of 44 LLMs with both Chain-of-Thought and Program-of-Thought prompting methods. Our experimental results reveal that the current best-performing system (i.e., GPT-4o) achieves only 60.9% accuracy using CoT prompting, leaving substantial room for improvement. Moreover, while augmenting LLMs with external knowledge can improve model performance (e.g., from 47.5% to 54.5% for Gemini-1.5-Pro), their accuracy remains significantly lower than the estimated human expert performance of 92%. We believe that FinanceMath can advance future research in the area of domain-specific knowledge retrieval and integration, particularly within the context of solving reasoning-intensive tasks.
Paper Structure (31 sections, 8 figures, 8 tables)

This paper contains 31 sections, 8 figures, 8 tables.

Table of Contents

  1. Introduction
  2. FinanceMath Benchmark
  3. Knowledge Bank Construction
  4. Knowledge Collection
  5. Automatic Knowledge Formulation
  6. Knowledge Bank Update and Maintenance
  7. FinanceMath Question Annotation
  8. Question Annotation
  9. Identifying Question-relevant Knowledge
  10. FinanceMath Solution Annotation
  11. Data Quality Validation
  12. Data Statistics and Dataset Release
  13. Human-level Performance Evaluation
  14. Evaluated Systems
  15. Large Language Models
  16. ...and 16 more sections

Figures (8)

  • Figure 1: An example of FinanceMath. To answer the given question, LLMs are required to comprehend specialized financial terms, such as "passive equity ownership interest" and "proportionate consolidation method". Additionally, they must interpret tabular data within the question and accurately identify question-relevant data points in the table.
  • Figure 2: An example of knowledge terms "Exchange Rate” included in the constructed knowledge bank.
  • Figure 3: Topic distribution of FinanceMath.
  • Figure 4: Example of zero-shot CoT prompt used.
  • Figure 5: Calibrated results of Chain-of-Thought prompting on the development set with an external calculator for math computation. Performing complex math computations correctly is still challenging for LLMs, especially open-source ones.
  • ...and 3 more figures