Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating LLMs' Mathematical Reasoning in Financial Document Question Answering

Pragya Srivastava, Manuj Malik, Vivek Gupta, Tanuja Ganu, Dan Roth

TL;DR

This work evaluates large language models on mathematical reasoning tasks over semi-structured financial documents, leveraging four finance-focused tabular datasets. It introduces the EEDP prompting paradigm (Elicit-Extract-Decompose-Predict) to decompose complex reasoning into atomic steps and demonstrates that EEDP can match or exceed program-based prompting methods while remaining computationally efficient. The study provides rich metadata annotations (reasoning steps, question types, table length, hierarchy depth, missing data) and a detailed error taxonomy for extraction, reasoning, and calculation, highlighting core weaknesses and guiding future improvements. Overall, the findings illuminate both the capabilities and limits of current LLMs in finance-domain numerical reasoning and offer practical prompting strategies to improve reliability in semi-structured document QA.

Abstract

Large Language Models (LLMs), excel in natural language understanding, but their capability for complex mathematical reasoning with an amalgamation of structured tables and unstructured text is uncertain. This study explores LLMs' mathematical reasoning on four financial tabular question-answering datasets: TATQA, FinQA, ConvFinQA, and Multihiertt. Through extensive experiments with various models and prompting techniques, we assess how LLMs adapt to complex tables and mathematical tasks. We focus on sensitivity to table complexity and performance variations with an increasing number of arithmetic reasoning steps. The results provide insights into LLMs' capabilities and limitations in handling complex mathematical scenarios for semi-structured tables. Ultimately, we introduce a novel prompting technique tailored to semi-structured documents, matching or outperforming other baselines in performance while providing a nuanced understanding of LLMs abilities for such a task.

Evaluating LLMs' Mathematical Reasoning in Financial Document Question Answering

TL;DR

This work evaluates large language models on mathematical reasoning tasks over semi-structured financial documents, leveraging four finance-focused tabular datasets. It introduces the EEDP prompting paradigm (Elicit-Extract-Decompose-Predict) to decompose complex reasoning into atomic steps and demonstrates that EEDP can match or exceed program-based prompting methods while remaining computationally efficient. The study provides rich metadata annotations (reasoning steps, question types, table length, hierarchy depth, missing data) and a detailed error taxonomy for extraction, reasoning, and calculation, highlighting core weaknesses and guiding future improvements. Overall, the findings illuminate both the capabilities and limits of current LLMs in finance-domain numerical reasoning and offer practical prompting strategies to improve reliability in semi-structured document QA.

Abstract

Large Language Models (LLMs), excel in natural language understanding, but their capability for complex mathematical reasoning with an amalgamation of structured tables and unstructured text is uncertain. This study explores LLMs' mathematical reasoning on four financial tabular question-answering datasets: TATQA, FinQA, ConvFinQA, and Multihiertt. Through extensive experiments with various models and prompting techniques, we assess how LLMs adapt to complex tables and mathematical tasks. We focus on sensitivity to table complexity and performance variations with an increasing number of arithmetic reasoning steps. The results provide insights into LLMs' capabilities and limitations in handling complex mathematical scenarios for semi-structured tables. Ultimately, we introduce a novel prompting technique tailored to semi-structured documents, matching or outperforming other baselines in performance while providing a nuanced understanding of LLMs abilities for such a task.
Paper Structure (40 sections, 2 equations, 18 figures, 3 tables)

This paper contains 40 sections, 2 equations, 18 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Metadata Annotations
  3. 1. Number of Reasoning Steps:
  4. 2. Question Categorization:
  5. 3. Table Length:
  6. 4. Hierarchical Complexity:
  7. 5. Missing Information:
  8. Annotation Splits.
  9. Experimental Results
  10. LLMs Prompting Methods:
  11. EEDP Prompting Strategy:
  12. Results and Analysis.
  13. EEDP's Computational Efficiency
  14. Where do LLMs fail?
  15. 1. Incorrect Extraction:
  16. ...and 25 more sections

Figures (18)

  • Figure 1: An example of a semi-structured financial document question answering.
  • Figure 2: A comparison showcasing the performance trends across various datasets with the increasing number of reasoning steps. The analysis contrasts the effectiveness of EEDP (our method) against PoT in addressing complex reasoning.
  • Figure 3: A comparison showcasing the performance trends observed in various datasets across different question types. The analysis contrasts the effectiveness of EEDP (our method) against Few-Shot PoT (PoT). Best viewed in color.
  • Figure 4: Our EEDP Approach (a.) Instructions, and (b.) Demonstration.
  • Figure 5: Accuracy of different arithmetic operations across different orders of magnitude.
  • ...and 13 more figures