Structure First, Reason Next: Enhancing a Large Language Model using Knowledge Graph for Numerical Reasoning in Financial Documents

TL;DR

This work tackles numerical reasoning in financial documents by augmenting large language models with structured knowledge graphs. It introduces an end-to-end pipeline that preprocesses documents, constructs a KG using a predefined financial schema, filters triplets with a lightweight MLP, and then reasons with an LLM over the filtered facts. Evaluated on the FinQA benchmark with Llama 3.1 8B Instruct, the KG-augmented approach yields a relative improvement of about 12.3% in execution accuracy over the vanilla Llama model, demonstrating the benefit of structural grounding for temporal and multi-hop numerical reasoning. The framework is designed to be adaptable to other financial datasets and supports clear separation of preprocessing, KG extraction, and reasoning components for scalable reasoning in finance.

Abstract

Numerical reasoning is an important task in the analysis of financial documents. It helps in understanding and performing numerical predictions with logical conclusions for the given query seeking answers from financial texts. Recently, Large Language Models (LLMs) have shown promising results in multiple Question-Answering (Q-A) systems with the capability of logical reasoning. As documents related to finance often consist of long and complex financial contexts, LLMs appear well-suited for building high-quality automated financial question-answering systems. However, LLMs often face challenges in accurately processing the various numbers within financial reports. Extracting numerical data from unstructured text and semi-structured tables, and reliably performing accurate calculations, remains a significant bottleneck for numerical reasoning in most state-of-the-art LLMs. Recent studies have shown that structured data augmentations, such as Knowledge Graphs (KGs), have notably improved the predictions of LLMs along with logical explanations. Thus, it is an important requirement to consider inherent structured information in financial reports while using LLMs for various financial analytics. This paper proposes a framework to incorporate structured information using KGs along with LLM predictions for numerical reasoning tasks. The KGs are extracted using a proposed schema inherently from the document under processing. We evaluated our proposed framework over the benchmark data FinQA, using an open-source LLM, namely Llama 3.1 8B Instruct. We observed that the proposed framework improved execution accuracy by approximately 12% relative to the vanilla LLM.

Figures (1)

• Figure 1: end-to-end pipeline for numerical reasoning in financial data with inherent knowledge graph and LLMs.