Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BankMathBench: A Benchmark for Numerical Reasoning in Banking Scenarios

Yunseung Lee, Subin Kim, Youngjun Kwak, Jaegul Choo

TL;DR

BankMathBench is proposed, a domain-specific dataset that reflects realistic banking tasks and exhibited notable improvements in both formula generation and numerical reasoning accuracy when trained on open-source LLMs, demonstrating the dataset's effectiveness in enhancing domain-specific reasoning.

Abstract

Large language models (LLMs)-based chatbots are increasingly being adopted in the financial domain, particularly in digital banking, to handle customer inquiries about products such as deposits, savings, and loans. However, these models still exhibit low accuracy in core banking computations-including total payout estimation, comparison of products with varying interest rates, and interest calculation under early repayment conditions. Such tasks require multi-step numerical reasoning and contextual understanding of banking products, yet existing LLMs often make systematic errors-misinterpreting product types, applying conditions incorrectly, or failing basic calculations involving exponents and geometric progressions. However, such errors have rarely been captured by existing benchmarks. Mathematical datasets focus on fundamental math problems, whereas financial benchmarks primarily target financial documents, leaving everyday banking scenarios underexplored. To address this limitation, we propose BankMathBench, a domain-specific dataset that reflects realistic banking tasks. BankMathBench is organized in three levels of difficulty-basic, intermediate, and advanced-corresponding to single-product reasoning, multi-product comparison, and multi-condition scenarios, respectively. When trained on BankMathBench, open-source LLMs exhibited notable improvements in both formula generation and numerical reasoning accuracy, demonstrating the dataset's effectiveness in enhancing domain-specific reasoning. With tool-augmented fine-tuning, the models achieved average accuracy increases of 57.6%p (basic), 75.1%p (intermediate), and 62.9%p (advanced), representing significant gains over zero-shot baselines. These findings highlight BankMathBench as a reliable benchmark for evaluating and advancing LLMs' numerical reasoning in real-world banking scenarios.

BankMathBench: A Benchmark for Numerical Reasoning in Banking Scenarios

TL;DR

BankMathBench is proposed, a domain-specific dataset that reflects realistic banking tasks and exhibited notable improvements in both formula generation and numerical reasoning accuracy when trained on open-source LLMs, demonstrating the dataset's effectiveness in enhancing domain-specific reasoning.

Abstract

Large language models (LLMs)-based chatbots are increasingly being adopted in the financial domain, particularly in digital banking, to handle customer inquiries about products such as deposits, savings, and loans. However, these models still exhibit low accuracy in core banking computations-including total payout estimation, comparison of products with varying interest rates, and interest calculation under early repayment conditions. Such tasks require multi-step numerical reasoning and contextual understanding of banking products, yet existing LLMs often make systematic errors-misinterpreting product types, applying conditions incorrectly, or failing basic calculations involving exponents and geometric progressions. However, such errors have rarely been captured by existing benchmarks. Mathematical datasets focus on fundamental math problems, whereas financial benchmarks primarily target financial documents, leaving everyday banking scenarios underexplored. To address this limitation, we propose BankMathBench, a domain-specific dataset that reflects realistic banking tasks. BankMathBench is organized in three levels of difficulty-basic, intermediate, and advanced-corresponding to single-product reasoning, multi-product comparison, and multi-condition scenarios, respectively. When trained on BankMathBench, open-source LLMs exhibited notable improvements in both formula generation and numerical reasoning accuracy, demonstrating the dataset's effectiveness in enhancing domain-specific reasoning. With tool-augmented fine-tuning, the models achieved average accuracy increases of 57.6%p (basic), 75.1%p (intermediate), and 62.9%p (advanced), representing significant gains over zero-shot baselines. These findings highlight BankMathBench as a reliable benchmark for evaluating and advancing LLMs' numerical reasoning in real-world banking scenarios.
Paper Structure (22 sections, 4 figures, 7 tables)

This paper contains 22 sections, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Numerical Reasoning with Large Language Models
  4. Numerical reasoning Benchmarks in Financial Domain
  5. Methodology
  6. Question Generation
  7. Basic Level
  8. Intermediate Level
  9. Advanced Level
  10. Data Augmentation
  11. Solution Generation and Data Filtering
  12. Reasoning Generation
  13. Expert Validation
  14. Experimental Design
  15. Results
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Examples of frequently asked customer queries in real banking branches.
  • Figure 2: Overview of the BankMathBench data generation pipeline, which comprises three stages: (a) question generation, (b) solution generation and automatic verification, and (c) reasoning generation.
  • Figure 3: Example of the question augmentation process.
  • Figure 4: Instruction format used across all models for zero-shot evaluation and supervised fine-tuning (SFT).