Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Program of Thoughts for Financial Reasoning: Leveraging Dynamic In-Context Examples and Generative Retrieval

Subhendu Khatuya, Shashwat Naidu, Pawan Goyal, Niloy Ganguly

TL;DR

This work tackles financial numerical reasoning by introducing FINDER, a two-step framework that first retrieves relevant facts from unstructured text and tables using an instruction-tuned generative retriever, then computes the answer with context-aware, dynamic Program of Thought prompting. The approach leverages LoRA-tuned FLAN-T5 for fact retrieval and a dynamic in-context exemplar strategy (via clustering) to steer PoT-based Python code generation, executed by an external solver (SymPy). FINDER achieves state-of-the-art execution accuracy on FinQA and ConvFinQA, with notable improvements over prior methods and strong parameter efficiency compared to large retriever models. The results are supported by extensive ablations showing the importance of targeted retrieval, dynamic exemplars, and modular design, and point to future work in external knowledge integration and broader domain deployment.

Abstract

Despite continuous advancements in the capabilities of large language models (LLMs), numerical reasoning remains a challenging area. Techniques like chain-of-thought prompting, tree-of-thought prompting, and program-of-thought prompting guide LLMs through intermediate reasoning steps. Although in-context learning with few-shot prompting has improved performance, LLMs still lag behind state-of-the-art models on financial numerical reasoning datasets such as FinQA and ConvFinQA. In this work, we introduce FINDER, a novel two-step framework, to enhance LLMs' capabilities in financial numerical reasoning. The first step utilizes a generative retriever to extract relevant facts from unstructured data, including both text and tables. This is followed by context-aware Program of Thought prompting with dynamic selection of in-context examples. Our model FINDER achieves a new state-of-the-art performance on both the FinQA and ConvFinQA datasets, surpassing previous benchmarks with execution accuracy improvements of 5.98% and 4.05%, respectively.

Program of Thoughts for Financial Reasoning: Leveraging Dynamic In-Context Examples and Generative Retrieval

TL;DR

This work tackles financial numerical reasoning by introducing FINDER, a two-step framework that first retrieves relevant facts from unstructured text and tables using an instruction-tuned generative retriever, then computes the answer with context-aware, dynamic Program of Thought prompting. The approach leverages LoRA-tuned FLAN-T5 for fact retrieval and a dynamic in-context exemplar strategy (via clustering) to steer PoT-based Python code generation, executed by an external solver (SymPy). FINDER achieves state-of-the-art execution accuracy on FinQA and ConvFinQA, with notable improvements over prior methods and strong parameter efficiency compared to large retriever models. The results are supported by extensive ablations showing the importance of targeted retrieval, dynamic exemplars, and modular design, and point to future work in external knowledge integration and broader domain deployment.

Abstract

Despite continuous advancements in the capabilities of large language models (LLMs), numerical reasoning remains a challenging area. Techniques like chain-of-thought prompting, tree-of-thought prompting, and program-of-thought prompting guide LLMs through intermediate reasoning steps. Although in-context learning with few-shot prompting has improved performance, LLMs still lag behind state-of-the-art models on financial numerical reasoning datasets such as FinQA and ConvFinQA. In this work, we introduce FINDER, a novel two-step framework, to enhance LLMs' capabilities in financial numerical reasoning. The first step utilizes a generative retriever to extract relevant facts from unstructured data, including both text and tables. This is followed by context-aware Program of Thought prompting with dynamic selection of in-context examples. Our model FINDER achieves a new state-of-the-art performance on both the FinQA and ConvFinQA datasets, surpassing previous benchmarks with execution accuracy improvements of 5.98% and 4.05%, respectively.
Paper Structure (31 sections, 2 equations, 8 figures, 8 tables)

This paper contains 31 sections, 2 equations, 8 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Task Formulation
  5. Relevant Fact Retriever
  6. Motivation
  7. Generative Retrieval
  8. Comparative Experiments:
  9. Target Answer Computation
  10. Dynamic In-Context Example Selection
  11. Enhancements Over Existing framework:
  12. Final Answer Computation
  13. Dataset & Evaluation Metrics
  14. Baselines
  15. Experimental Setup
  16. ...and 16 more sections

Figures (8)

  • Figure 1: Architecture diagram of our proposed framework FINDER. FLAN-T5 processes task-specific instructions, context (text and tables), and questions to generate relevant facts. GPT-4 generates PoT style code using a Final Prompt with in-context examples from trained PromptPG and retrieved relevant facts. The SymPy executor then interprets this code to produce the final answer.
  • Figure 2: In-Context example formatted in Program of Thought Prompt style for FinQA.
  • Figure 3: t-SNE plot illustrates the clusters formed from the embeddings of the training questions, showcasing a subset of clusters for visualization purposes. Each cluster represents a theme e.g. growth rates, amortization, percentage change, ROI, etc.
  • Figure 4: Few shot exemplar analysis of FinQA and ConvFinQA
  • Figure 5: Fact Retrieval Errors: The Retriever module failed to retrieve the relevant facts necessary for the question.
  • ...and 3 more figures