Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Weakly Supervised Veracity Classification with LLM-Predicted Credibility Signals

João A. Leite, Olesya Razuvayevskaya, Kalina Bontcheva, Carolina Scarton

TL;DR

Pastel is introduced, a weakly supervised approach that leverages large language models (LLMs) to extract credibility signals from web content, and subsequently combines them to predict the veracity of content without relying on human supervision.

Abstract

Credibility signals represent a wide range of heuristics typically used by journalists and fact-checkers to assess the veracity of online content. Automating the extraction of credibility signals presents significant challenges due to the necessity of training high-accuracy, signal-specific extractors, coupled with the lack of sufficiently large annotated datasets. This paper introduces Pastel (Prompted weAk Supervision wiTh crEdibility signaLs), a weakly supervised approach that leverages large language models (LLMs) to extract credibility signals from web content, and subsequently combines them to predict the veracity of content without relying on human supervision. We validate our approach using four article-level misinformation detection datasets, demonstrating that Pastel outperforms zero-shot veracity detection by 38.3% and achieves 86.7% of the performance of the state-of-the-art system trained with human supervision. Moreover, in cross-domain settings where training and testing datasets originate from different domains, Pastel significantly outperforms the state-of-the-art supervised model by 63%. We further study the association between credibility signals and veracity, and perform an ablation study showing the impact of each signal on model performance. Our findings reveal that 12 out of the 19 proposed signals exhibit strong associations with veracity across all datasets, while some signals show domain-specific strengths.

Weakly Supervised Veracity Classification with LLM-Predicted Credibility Signals

TL;DR

Pastel is introduced, a weakly supervised approach that leverages large language models (LLMs) to extract credibility signals from web content, and subsequently combines them to predict the veracity of content without relying on human supervision.

Abstract

Credibility signals represent a wide range of heuristics typically used by journalists and fact-checkers to assess the veracity of online content. Automating the extraction of credibility signals presents significant challenges due to the necessity of training high-accuracy, signal-specific extractors, coupled with the lack of sufficiently large annotated datasets. This paper introduces Pastel (Prompted weAk Supervision wiTh crEdibility signaLs), a weakly supervised approach that leverages large language models (LLMs) to extract credibility signals from web content, and subsequently combines them to predict the veracity of content without relying on human supervision. We validate our approach using four article-level misinformation detection datasets, demonstrating that Pastel outperforms zero-shot veracity detection by 38.3% and achieves 86.7% of the performance of the state-of-the-art system trained with human supervision. Moreover, in cross-domain settings where training and testing datasets originate from different domains, Pastel significantly outperforms the state-of-the-art supervised model by 63%. We further study the association between credibility signals and veracity, and perform an ablation study showing the impact of each signal on model performance. Our findings reveal that 12 out of the 19 proposed signals exhibit strong associations with veracity across all datasets, while some signals show domain-specific strengths.
Paper Structure (22 sections, 6 equations, 5 figures, 7 tables)

This paper contains 22 sections, 6 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Article-level Veracity Classification
  4. Credibility Signals
  5. Veracity Classification with Weak Supervision
  6. Prompted weAk Supervision wiTh crEdibility signaLs (PASTEL)
  7. Credibility Signals Considered
  8. Signal Extraction (LLM Prompting)
  9. Weak Supervision
  10. Experimental Setup
  11. Datasets
  12. Evaluation
  13. Large Language Model
  14. Baselines
  15. Results
  16. ...and 7 more sections

Figures (5)

  • Figure 1: Illustration of PASTEL.
  • Figure 2: Prompt template to extract credibility signals.
  • Figure 3: Mean confusion matrices obtained with Pastel. Means and standard deviations reported across 10-fold cross-validation. Labels $0$ and $1$ refer to non-misinformation and misinformation, respectivelly.
  • Figure 4: Distribution of LLM responses per credibility signal for non-misinformation articles (solid bars) and misinformation articles (hashed bars) averaged across all datasets.
  • Figure 5: Normalised Pearson's $\chi^2$ statistics per credibility signal. Credibility signals where the null hypothesis $H_0$ is rejected ($p<0.05$, $1$ degree of freedom) are marked with an asterisk ($\ast$). Results are shown for each dataset, and aggregated by domain; 'Politics' displays the average of FakeNewsAMT and PolitiFact, and 'Entertainment' shows the average of Celebrity and GossipCop. All four datasets are averaged into 'All'. For aggregate results, we reject $H_0$ if $H_0$ is rejected in all the aggregated datasets. Credibility signals are sorted in descending order based on the overall average ('All').