Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FinReflectKG -- MultiHop: Financial QA Benchmark for Reasoning with Knowledge Graph Evidence

Abhinav Arun, Reetu Raj Harsh, Bhaskarjit Sarmah, Stefano Pasquali

TL;DR

FinReflectKG - MultiHop addresses difficulty of multi-hop financial QA by grounding retrieval in a temporally indexed financial knowledge graph derived from SEC 10-K filings. It introduces a pattern-based QA generation workflow, KG-driven chunk identification, and a multi-criteria annotation process, enabling rigorous ground-truth evidence paths. The benchmark compares KG-grounded evidence against text-window and distractor-rich retrieval, showing KG-guided retrieval improves correctness by about 24% and reduces input token usage by about 84.5% across open-source and proprietary models. The dataset includes a curated subset of 555 QA pairs spanning intra-document, inter-year, and cross-company reasoning, designed to catalyze research on efficient, trustworthy financial QA.

Abstract

Multi-hop reasoning over financial disclosures is often a retrieval problem before it becomes a reasoning or generation problem: relevant facts are dispersed across sections, filings, companies, and years, and LLMs often expend excessive tokens navigating noisy context. Without precise Knowledge Graph (KG)-guided selection of relevant context, even strong reasoning models either fail to answer or consume excessive tokens, whereas KG-linked evidence enables models to focus their reasoning on composing already retrieved facts. We present FinReflectKG - MultiHop, a benchmark built on FinReflectKG, a temporally indexed financial KG that links audited triples to source chunks from S&P 100 filings (2022-2024). Mining frequent 2-3 hop subgraph patterns across sectors (via GICS taxonomy), we generate financial analyst style questions with exact supporting evidence from the KG. A two-phase pipeline first creates QA pairs via pattern-specific prompts, followed by a multi-criteria quality control evaluation to ensure QA validity. We then evaluate three controlled retrieval scenarios: (S1) precise KG-linked paths; (S2) text-only page windows centered on relevant text spans; and (S3) relevant page windows with randomizations and distractors. Across both reasoning and non-reasoning models, KG-guided precise retrieval yields substantial gains on the FinReflectKG - MultiHop QA benchmark dataset, boosting correctness scores by approximately 24 percent while reducing token utilization by approximately 84.5 percent compared to the page window setting, which reflects the traditional vector retrieval paradigm. Spanning intra-document, inter-year, and cross-company scopes, our work underscores the pivotal role of knowledge graphs in efficiently connecting evidence for multi-hop financial QA. We also release a curated subset of the benchmark (555 QA Pairs) to catalyze further research.

FinReflectKG -- MultiHop: Financial QA Benchmark for Reasoning with Knowledge Graph Evidence

TL;DR

FinReflectKG - MultiHop addresses difficulty of multi-hop financial QA by grounding retrieval in a temporally indexed financial knowledge graph derived from SEC 10-K filings. It introduces a pattern-based QA generation workflow, KG-driven chunk identification, and a multi-criteria annotation process, enabling rigorous ground-truth evidence paths. The benchmark compares KG-grounded evidence against text-window and distractor-rich retrieval, showing KG-guided retrieval improves correctness by about 24% and reduces input token usage by about 84.5% across open-source and proprietary models. The dataset includes a curated subset of 555 QA pairs spanning intra-document, inter-year, and cross-company reasoning, designed to catalyze research on efficient, trustworthy financial QA.

Abstract

Multi-hop reasoning over financial disclosures is often a retrieval problem before it becomes a reasoning or generation problem: relevant facts are dispersed across sections, filings, companies, and years, and LLMs often expend excessive tokens navigating noisy context. Without precise Knowledge Graph (KG)-guided selection of relevant context, even strong reasoning models either fail to answer or consume excessive tokens, whereas KG-linked evidence enables models to focus their reasoning on composing already retrieved facts. We present FinReflectKG - MultiHop, a benchmark built on FinReflectKG, a temporally indexed financial KG that links audited triples to source chunks from S&P 100 filings (2022-2024). Mining frequent 2-3 hop subgraph patterns across sectors (via GICS taxonomy), we generate financial analyst style questions with exact supporting evidence from the KG. A two-phase pipeline first creates QA pairs via pattern-specific prompts, followed by a multi-criteria quality control evaluation to ensure QA validity. We then evaluate three controlled retrieval scenarios: (S1) precise KG-linked paths; (S2) text-only page windows centered on relevant text spans; and (S3) relevant page windows with randomizations and distractors. Across both reasoning and non-reasoning models, KG-guided precise retrieval yields substantial gains on the FinReflectKG - MultiHop QA benchmark dataset, boosting correctness scores by approximately 24 percent while reducing token utilization by approximately 84.5 percent compared to the page window setting, which reflects the traditional vector retrieval paradigm. Spanning intra-document, inter-year, and cross-company scopes, our work underscores the pivotal role of knowledge graphs in efficiently connecting evidence for multi-hop financial QA. We also release a curated subset of the benchmark (555 QA Pairs) to catalyze further research.

Paper Structure

This paper contains 11 sections, 2 figures, 11 tables.

Table of Contents

  1. Introduction & Motivation
  2. Related Work
  3. Dataset Design: FinReflectKG - MultiHop
  4. Financial Analyst-Focused Pattern Generation
  5. Knowledge Graph-Driven Chunk Identification
  6. Question & Answer Generation
  7. Annotation Schema & Process
  8. Experimental Design & Evaluation Protocol
  9. Experimental Results & Analysis
  10. Conclusion & Future Work
  11. Appendix

Figures (2)

  • Figure 1: Interactive labeling interface used for manual verification of reliability, groundedness, and relevance in multi-hop financial QA. The interface highlights how evidence is linked across multiple segments and disclosures, also requiring reasoning over financial relationships for a sample intra-document multihop question.
  • Figure 2: Additional view of the labeling tool showing source content, extracted triples, and reasoning patterns for connected context as in Figure \ref{['fig:sample_question_2nd_chunk']}. This illustrates the ongoing effort to build a cleaner, larger, and manually verified multi-hop financial QA dataset.