Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Belief-Sim: Towards Belief-Driven Simulation of Demographic Misinformation Susceptibility

Angana Borah, Zohaib Khan, Rada Mihalcea, Verónica Pérez-Rosas

TL;DR

It is shown that beliefs provide a strong prior for simulating misinformation susceptibility, with accuracy up to 92% and a multi-fold evaluation using: susceptibility accuracy and counterfactual demographic sensitivity.

Abstract

Misinformation is a growing societal threat, and susceptibility to misinformative claims varies across demographic groups due to differences in underlying beliefs. As Large Language Models (LLMs) are increasingly used to simulate human behaviors, we investigate whether they can simulate demographic misinformation susceptibility, treating beliefs as a primary driving factor. We introduce BeliefSim, a simulation framework that constructs demographic belief profiles using psychology-informed taxonomies and survey priors. We study prompt-based conditioning and post-training adaptation, and conduct a multi-fold evaluation using: (i) susceptibility accuracy and (ii) counterfactual demographic sensitivity. Across both datasets and modeling strategies, we show that beliefs provide a strong prior for simulating misinformation susceptibility, with accuracy up to 92%.

Belief-Sim: Towards Belief-Driven Simulation of Demographic Misinformation Susceptibility

TL;DR

It is shown that beliefs provide a strong prior for simulating misinformation susceptibility, with accuracy up to 92% and a multi-fold evaluation using: susceptibility accuracy and counterfactual demographic sensitivity.

Abstract

Misinformation is a growing societal threat, and susceptibility to misinformative claims varies across demographic groups due to differences in underlying beliefs. As Large Language Models (LLMs) are increasingly used to simulate human behaviors, we investigate whether they can simulate demographic misinformation susceptibility, treating beliefs as a primary driving factor. We introduce BeliefSim, a simulation framework that constructs demographic belief profiles using psychology-informed taxonomies and survey priors. We study prompt-based conditioning and post-training adaptation, and conduct a multi-fold evaluation using: (i) susceptibility accuracy and (ii) counterfactual demographic sensitivity. Across both datasets and modeling strategies, we show that beliefs provide a strong prior for simulating misinformation susceptibility, with accuracy up to 92%.
Paper Structure (34 sections, 1 equation, 11 figures, 10 tables)

This paper contains 34 sections, 1 equation, 11 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Belief Taxonomy
  4. Simulation Data
  5. Evaluation Data
  6. Belief Data
  7. Prompt-based Conditioning for Susceptibility Simulation
  8. Results
  9. Demographic-based Counterfactual Evaluation
  10. Post-training Adaptation for Susceptibility Simulation
  11. Results
  12. Lessons Learned and Actionable Steps
  13. Conclusion
  14. Taxonomy Dimensions and Imputed Beliefs
  15. Taxonomy formation
  16. ...and 19 more sections

Figures (11)

  • Figure 1: BeliefSim Framework: (1) Participant Data, Observed and Imputed Beliefs (based on Belief Taxonomy) are collected from surveys, (2) Methods consist of prompt-conditioning and post-training adaptation and (3) Evaluation using Susceptibility Accuracy, Counterfactual and Thematic Analysis.
  • Figure 2: Simulation Data Example.
  • Figure 3: Susceptibility Accuracies by settings and datasets, averaged across demographic groups and models. Imputed + Demo(graphic) (best) performs the best, with all belief included settings doing better than zero-shot and demo-only.
  • Figure 4: Demographic-based Counterfactual Eval Framework: (1) Utility of demographics, (2) Shortcut Reliance, and (3) Complementarity. Note that we perform a demographic swap within the same demographic group, i.e., swap male with female, rural with urban, etc. keeping the claim constant.
  • Figure 5: Flip-Rates for Counterfactual Evaluation: qwen and mistral models have lower flip-rates in comparison to llama.
  • ...and 6 more figures