Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FAITH: A Framework for Assessing Intrinsic Tabular Hallucinations in Finance

Mengao Zhang, Jiayu Fu, Tanya Warrier, Yuwen Wang, Tianhui Tan, Ke-wei Huang

TL;DR

The paper addresses intrinsic tabular hallucinations in finance by introducing FAITH, a scalable framework that constructs context-aware masked-span evaluation tasks from real financial documents to assess numeric grounding accuracy. It defines four financial reasoning types, develops robust numeric and unit matching protocols, and builds an MD&A-based dataset from S&P 500 10-K filings for evaluation. Empirical results reveal a clear hierarchy of model reliability, with top proprietary systems achieving high accuracy but still struggling on multi-step calculations, underscoring the need for domain-specific evaluation to ensure trustworthy financial Generative AI. FAITH thus offers a practical pathway to safer deployment of financial LLMs by enabling rigorous evaluation and guiding future improvements in numerical reasoning and grounding.

Abstract

Hallucination remains a critical challenge for deploying Large Language Models (LLMs) in finance. Accurate extraction and precise calculation from tabular data are essential for reliable financial analysis, since even minor numerical errors can undermine decision-making and regulatory compliance. Financial applications have unique requirements, often relying on context-dependent, numerical, and proprietary tabular data that existing hallucination benchmarks rarely capture. In this study, we develop a rigorous and scalable framework for evaluating intrinsic hallucinations in financial LLMs, conceptualized as a context-aware masked span prediction task over real-world financial documents. Our main contributions are: (1) a novel, automated dataset creation paradigm using a masking strategy; (2) a new hallucination evaluation dataset derived from S&P 500 annual reports; and (3) a comprehensive evaluation of intrinsic hallucination patterns in state-of-the-art LLMs on financial tabular data. Our work provides a robust methodology for in-house LLM evaluation and serves as a critical step toward building more trustworthy and reliable financial Generative AI systems.

FAITH: A Framework for Assessing Intrinsic Tabular Hallucinations in Finance

TL;DR

The paper addresses intrinsic tabular hallucinations in finance by introducing FAITH, a scalable framework that constructs context-aware masked-span evaluation tasks from real financial documents to assess numeric grounding accuracy. It defines four financial reasoning types, develops robust numeric and unit matching protocols, and builds an MD&A-based dataset from S&P 500 10-K filings for evaluation. Empirical results reveal a clear hierarchy of model reliability, with top proprietary systems achieving high accuracy but still struggling on multi-step calculations, underscoring the need for domain-specific evaluation to ensure trustworthy financial Generative AI. FAITH thus offers a practical pathway to safer deployment of financial LLMs by enabling rigorous evaluation and guiding future improvements in numerical reasoning and grounding.

Abstract

Hallucination remains a critical challenge for deploying Large Language Models (LLMs) in finance. Accurate extraction and precise calculation from tabular data are essential for reliable financial analysis, since even minor numerical errors can undermine decision-making and regulatory compliance. Financial applications have unique requirements, often relying on context-dependent, numerical, and proprietary tabular data that existing hallucination benchmarks rarely capture. In this study, we develop a rigorous and scalable framework for evaluating intrinsic hallucinations in financial LLMs, conceptualized as a context-aware masked span prediction task over real-world financial documents. Our main contributions are: (1) a novel, automated dataset creation paradigm using a masking strategy; (2) a new hallucination evaluation dataset derived from S&P 500 annual reports; and (3) a comprehensive evaluation of intrinsic hallucination patterns in state-of-the-art LLMs on financial tabular data. Our work provides a robust methodology for in-house LLM evaluation and serves as a critical step toward building more trustworthy and reliable financial Generative AI systems.

Paper Structure

This paper contains 30 sections, 1 equation, 1 figure, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Hallucination Evaluation Datasets
  4. Tabular Reasoning and Evaluation
  5. Tabular Reasoning and Evaluation
  6. Automatic Dataset Construction
  7. Task Definition
  8. Problem Formulation
  9. Masking Criteria
  10. Robust Evaluation
  11. Numeric Matching.
  12. Unit Matching.
  13. Financial Reasoning Scenarios
  14. Pilot Study
  15. Evaluation Dataset
  16. ...and 15 more sections

Figures (1)

  • Figure 1: Illustration of the task definition and data processing.