Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fin-Fact: A Benchmark Dataset for Multimodal Financial Fact Checking and Explanation Generation

Aman Rangapur, Haoran Wang, Ling Jian, Kai Shu

TL;DR

Fin-Fact introduces a finance-focused multimodal fact-checking benchmark with expert annotations and explanations. It combines text and images across true/false/NEI labels to evaluate multimodal models and NLI, revealing the substantial influence of visuals and the need for robust explanation generation. Key findings point to GPT-4V's strong NLI performance and variability in explanation quality, underscoring the dataset's role as a baseline for reproducible, transparent financial misinformation research. The work provides open-source code and a pathway toward expanding multimedia content for more comprehensive verification in finance.

Abstract

Fact-checking in financial domain is under explored, and there is a shortage of quality dataset in this domain. In this paper, we propose Fin-Fact, a benchmark dataset for multimodal fact-checking within the financial domain. Notably, it includes professional fact-checker annotations and justifications, providing expertise and credibility. With its multimodal nature encompassing both textual and visual content, Fin-Fact provides complementary information sources to enhance factuality analysis. Its primary objective is combating misinformation in finance, fostering transparency, and building trust in financial reporting and news dissemination. By offering insightful explanations, Fin-Fact empowers users, including domain experts and end-users, to understand the reasoning behind fact-checking decisions, validating claim credibility, and fostering trust in the fact-checking process. The Fin-Fact dataset, along with our experimental codes is available at https://github.com/IIT-DM/Fin-Fact/.

Fin-Fact: A Benchmark Dataset for Multimodal Financial Fact Checking and Explanation Generation

TL;DR

Fin-Fact introduces a finance-focused multimodal fact-checking benchmark with expert annotations and explanations. It combines text and images across true/false/NEI labels to evaluate multimodal models and NLI, revealing the substantial influence of visuals and the need for robust explanation generation. Key findings point to GPT-4V's strong NLI performance and variability in explanation quality, underscoring the dataset's role as a baseline for reproducible, transparent financial misinformation research. The work provides open-source code and a pathway toward expanding multimedia content for more comprehensive verification in finance.

Abstract

Fact-checking in financial domain is under explored, and there is a shortage of quality dataset in this domain. In this paper, we propose Fin-Fact, a benchmark dataset for multimodal fact-checking within the financial domain. Notably, it includes professional fact-checker annotations and justifications, providing expertise and credibility. With its multimodal nature encompassing both textual and visual content, Fin-Fact provides complementary information sources to enhance factuality analysis. Its primary objective is combating misinformation in finance, fostering transparency, and building trust in financial reporting and news dissemination. By offering insightful explanations, Fin-Fact empowers users, including domain experts and end-users, to understand the reasoning behind fact-checking decisions, validating claim credibility, and fostering trust in the fact-checking process. The Fin-Fact dataset, along with our experimental codes is available at https://github.com/IIT-DM/Fin-Fact/.
Paper Structure (17 sections, 7 figures, 6 tables)

This paper contains 17 sections, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Fin-Fact Dataset
  4. Data Collection and Preprocessing
  5. Dataset Statistics
  6. Evaluation and Analysis
  7. RQ1: Multi modal performance
  8. RQ2: NLI performance
  9. RQ3: Explaination Generation
  10. Limitations
  11. Conclusion and Future Work
  12. Appendix
  13. Additional Details of Dataset
  14. Additional Evaluation Details
  15. Evaluation Setup and Reproducibility
  16. ...and 2 more sections

Figures (7)

  • Figure 1: Illustration of comprehensive multimodal fact-checking, including True, False, and Not Enough Information (NEI), alongside the creation of explanations.
  • Figure 2: Diverse sectors within the Fin-Fact dataset.
  • Figure 3: Example demonstration of GPT-4V for prediction and explanation generation of real-world claims.
  • Figure 4: Confidence score and it's calibration
  • Figure 5: Example of model-generated explanation as compared to the gold standard from Fin-Fact dataset.
  • ...and 2 more figures