Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

All That Glisters Is Not Gold: A Benchmark for Reference-Free Counterfactual Financial Misinformation Detection

Yuechen Jiang, Zhiwei Liu, Yupeng Cao, Yueru He, Chen Xu, Ziyang Xu, Zhiyang Deng, Prayag Tiwari, Xi Chen, Alejandro Lopez-Lira, Jimin Huang, Junichi Tsujii, Sophia Ananiadou

TL;DR

RFC-BENCH introduces a paragraph-level, reference-free benchmark for financial misinformation with minimally perturbed original–perturbed paragraph pairs across four manipulation types. It formalizes two evaluation tasks—Reference-free Detection and Comparative Diagnosis—and evaluates 14 LLMs, revealing a persistent accommodation-first bias: models struggle to flag manipulated paragraphs in isolation but perform well when an explicit original is provided for comparison. The dataset integrates rigorous data curation, expert validation, and dual-annotator reliability, yielding 1,845 retained instances and a hard-case subset for robustness analysis. The work highlights significant gaps in reference-free grounding for financial misinformation and provides a structured testbed to drive more reliable, grounded LLM behavior in finance.

Abstract

We introduce RFC Bench, a benchmark for evaluating large language models on financial misinformation under realistic news. RFC Bench operates at the paragraph level and captures the contextual complexity of financial news where meaning emerges from dispersed cues. The benchmark defines two complementary tasks: reference free misinformation detection and comparison based diagnosis using paired original perturbed inputs. Experiments reveal a consistent pattern: performance is substantially stronger when comparative context is available, while reference free settings expose significant weaknesses, including unstable predictions and elevated invalid outputs. These results indicate that current models struggle to maintain coherent belief states without external grounding. By highlighting this gap, RFC Bench provides a structured testbed for studying reference free reasoning and advancing more reliable financial misinformation detection in real world settings.

All That Glisters Is Not Gold: A Benchmark for Reference-Free Counterfactual Financial Misinformation Detection

TL;DR

RFC-BENCH introduces a paragraph-level, reference-free benchmark for financial misinformation with minimally perturbed original–perturbed paragraph pairs across four manipulation types. It formalizes two evaluation tasks—Reference-free Detection and Comparative Diagnosis—and evaluates 14 LLMs, revealing a persistent accommodation-first bias: models struggle to flag manipulated paragraphs in isolation but perform well when an explicit original is provided for comparison. The dataset integrates rigorous data curation, expert validation, and dual-annotator reliability, yielding 1,845 retained instances and a hard-case subset for robustness analysis. The work highlights significant gaps in reference-free grounding for financial misinformation and provides a structured testbed to drive more reliable, grounded LLM behavior in finance.

Abstract

We introduce RFC Bench, a benchmark for evaluating large language models on financial misinformation under realistic news. RFC Bench operates at the paragraph level and captures the contextual complexity of financial news where meaning emerges from dispersed cues. The benchmark defines two complementary tasks: reference free misinformation detection and comparison based diagnosis using paired original perturbed inputs. Experiments reveal a consistent pattern: performance is substantially stronger when comparative context is available, while reference free settings expose significant weaknesses, including unstable predictions and elevated invalid outputs. These results indicate that current models struggle to maintain coherent belief states without external grounding. By highlighting this gap, RFC Bench provides a structured testbed for studying reference free reasoning and advancing more reliable financial misinformation detection in real world settings.
Paper Structure (122 sections, 17 equations, 15 figures, 8 tables)

This paper contains 122 sections, 17 equations, 15 figures, 8 tables.

Table of Contents

  1. Introduction
  2. R
  3. Task Formulation
  4. Data Curation
  5. Data Acquisition
  6. Category-specific LLM Rewriting and Prompt Refinement
  7. Human Quality Assurance
  8. Expert Review and Audit
  9. Dual Annotator Evaluation and Reliability Analysis
  10. Dataset Finalization
  11. Evaluation
  12. Experiments
  13. Main Results
  14. Task 1: Reference-free Detection
  15. Task 2: Comparative Diagnosis
  16. ...and 107 more sections

Figures (15)

  • Figure 1: Counterfactual financial misinformation generated via minimal yet belief-shifting edits.
  • Figure 2: Overview of the RFC-BENCH212,175,55120,130,140 construction and evaluation workflow. A detailed, step-by-step description of the dataset construction workflow is provided in Appendix \ref{['app:RFC-data-workflow']} Figure \ref{['fig:workflow']}.
  • Figure 3: Task 1: Reference-free Detection (normalized radar chart).
  • Figure 4: Task 2: Comparative Diagnosis (normalized radar chart).
  • Figure 5: Accuracy trends on Task 1 under zero-shot, two-shot, and eight-shot settings. Few-shot prompting provides limited gains and fails to bridge the gap with pairwise evaluation.
  • ...and 10 more figures