Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Same Claim, Different Judgment: Benchmarking Scenario-Induced Bias in Multilingual Financial Misinformation Detection

Zhiwei Liu, Yupen Cao, Yuechen Jiang, Mohsinul Kabir, Polydoros Giannouris, Chen Xu, Ziyang Xu, Tianlei Zhu, Tariquzzaman Faisal, Triantafillos Papadopoulos, Yan Wang, Lingfei Qian, Xueqing Peng, Zhuohan Xie, Ye Yuan, Saeed Almheiri, Abdulrazzaq Alnajjar, Mingbin Chen, Harry Stuart, Paul Thompson, Prayag Tiwari, Alejandro Lopez-Lira, Xue Liu, Jimin Huang, Sophia Ananiadou

TL;DR

This work introduces MFMD-Scen52,84,110170,46,74, a scenario-conditioned, multilingual benchmark for financial misinformation detection that varies context along three axes: persona, region, and identity. By constructing a 502-item multilingual dataset (English, Chinese, Greek, Bengali) and evaluating 22 LLMs, the study reveals pronounced behavioral biases that shift decision boundaries, are magnified in low-resource languages, and depend on complex interactions between role and identity. The framework provides both standard detection metrics and a bias estimate (AM and MAV of F1 differences) to quantify context effects, offering a rigorous basis for bias mitigation in high-stakes financial misinformation tasks. The results have practical implications for the deployment of LLMs in finance, regulators, and researchers seeking robust, fair multilingual misinformation detection. The authors also share a public repository to enable ongoing benchmarking and improvement in bias mitigation.

Abstract

Large language models (LLMs) have been widely applied across various domains of finance. Since their training data are largely derived from human-authored corpora, LLMs may inherit a range of human biases. Behavioral biases can lead to instability and uncertainty in decision-making, particularly when processing financial information. However, existing research on LLM bias has mainly focused on direct questioning or simplified, general-purpose settings, with limited consideration of the complex real-world financial environments and high-risk, context-sensitive, multilingual financial misinformation detection tasks (\mfmd). In this work, we propose \mfmdscen, a comprehensive benchmark for evaluating behavioral biases of LLMs in \mfmd across diverse economic scenarios. In collaboration with financial experts, we construct three types of complex financial scenarios: (i) role- and personality-based, (ii) role- and region-based, and (iii) role-based scenarios incorporating ethnicity and religious beliefs. We further develop a multilingual financial misinformation dataset covering English, Chinese, Greek, and Bengali. By integrating these scenarios with misinformation claims, \mfmdscen enables a systematic evaluation of 22 mainstream LLMs. Our findings reveal that pronounced behavioral biases persist across both commercial and open-source models. This project will be available at https://github.com/lzw108/FMD.

Same Claim, Different Judgment: Benchmarking Scenario-Induced Bias in Multilingual Financial Misinformation Detection

TL;DR

This work introduces MFMD-Scen52,84,110170,46,74, a scenario-conditioned, multilingual benchmark for financial misinformation detection that varies context along three axes: persona, region, and identity. By constructing a 502-item multilingual dataset (English, Chinese, Greek, Bengali) and evaluating 22 LLMs, the study reveals pronounced behavioral biases that shift decision boundaries, are magnified in low-resource languages, and depend on complex interactions between role and identity. The framework provides both standard detection metrics and a bias estimate (AM and MAV of F1 differences) to quantify context effects, offering a rigorous basis for bias mitigation in high-stakes financial misinformation tasks. The results have practical implications for the deployment of LLMs in finance, regulators, and researchers seeking robust, fair multilingual misinformation detection. The authors also share a public repository to enable ongoing benchmarking and improvement in bias mitigation.

Abstract

Large language models (LLMs) have been widely applied across various domains of finance. Since their training data are largely derived from human-authored corpora, LLMs may inherit a range of human biases. Behavioral biases can lead to instability and uncertainty in decision-making, particularly when processing financial information. However, existing research on LLM bias has mainly focused on direct questioning or simplified, general-purpose settings, with limited consideration of the complex real-world financial environments and high-risk, context-sensitive, multilingual financial misinformation detection tasks (\mfmd). In this work, we propose \mfmdscen, a comprehensive benchmark for evaluating behavioral biases of LLMs in \mfmd across diverse economic scenarios. In collaboration with financial experts, we construct three types of complex financial scenarios: (i) role- and personality-based, (ii) role- and region-based, and (iii) role-based scenarios incorporating ethnicity and religious beliefs. We further develop a multilingual financial misinformation dataset covering English, Chinese, Greek, and Bengali. By integrating these scenarios with misinformation claims, \mfmdscen enables a systematic evaluation of 22 mainstream LLMs. Our findings reveal that pronounced behavioral biases persist across both commercial and open-source models. This project will be available at https://github.com/lzw108/FMD.
Paper Structure (44 sections, 3 equations, 7 figures, 15 tables)

This paper contains 44 sections, 3 equations, 7 figures, 15 tables.

Table of Contents

  1. Introduction
  2. MFMD-Scen52,84,110170,46,74 Benchmark
  3. Task Formulation
  4. Task 1: Detection in Different Personality Scenarios (MFMD52,84,110170,46,74-persona)
  5. Task 2: Detection in Different Financial Markets (MFMD52,84,110170,46,74-region)
  6. Task 3: Detection in Different Identities (MFMD52,84,110170,46,74-identity)
  7. Multilingual Financial Misinformation Dataset Construction
  8. Evaluations
  9. Models
  10. Evaluation metrics
  11. Human-level Performance Measurement
  12. Evaluation Results
  13. Main findings
  14. Results on MFMD52,84,110170,46,74-persona
  15. Results on MFMD52,84,110170,46,74-region
  16. ...and 29 more sections

Figures (7)

  • Figure 1: An example when an LLM detects financial misinformation in a different scenario.
  • Figure 2: Overview of MFMD-Scen52,84,110170,46,74 Benchmark. The upper part is the three subtasks of financial scenarios. MFMD52,84,110170,46,74-persona: personality scenarios based on roles and behavioral finance biases. MFMD52,84,110170,46,74-region: scenarios based on the roles of different financial regions. MFMD52,84,110170,46,74-identity: scenarios based on roles and different ethnicities and Faith. The second is the financial misinformation dataset (Sec \ref{['sec:MFMDclaims']}). Combine the scenarios and misinformation claims to obtain the MFMD-Scen52,84,110170,46,74.
  • Figure 3: Radar chart on MFMD52,84,110170,46,74-persona. The arithmetic mean (AM) and the mean absolute values (MAV) across 22 models of F1 in MFMD52,84,110170,46,74-persona. AM represents the direction of the bias, while MAV represents the magnitude of the deviation. In the legend, the dark color represents the False category, while the corresponding light color represents the True category.
  • Figure 4: Results of some representative LLMs, and the AM, MAV across 22 models of F1 in MFMD52,84,110170,46,74-region. The dashed line represents the base behavior without a scenario. "AP": Asia Pacific. "CM": China Mainland.
  • Figure 5: Bias of some representative LLMs, and the AM, MAV across 22 models of F1 in MFMD52,84,110170,46,74-identity. "RI": Retail Investor. "CO": Company Owner.
  • ...and 2 more figures