Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CoALFake: Collaborative Active Learning with Human-LLM Co-Annotation for Cross-Domain Fake News Detection

Esma Aïmeur, Gilles Brassard, Dorsaf Sallami

Abstract

The proliferation of fake news across diverse domains highlights critical limitations in current detection systems, which often exhibit narrow domain specificity and poor generalization. Existing cross-domain approaches face two key challenges: (1) reliance on labelled data, which is frequently unavailable and resource intensive to acquire and (2) information loss caused by rigid domain categorization or neglect of domain-specific features. To address these issues, we propose CoALFake, a novel approach for cross-domain fake news detection that integrates Human-Large Language Model (LLM) co-annotation with domain-aware Active Learning (AL). Our method employs LLMs for scalable, low-cost annotation while maintaining human oversight to ensure label reliability. By integrating domain embedding techniques, the CoALFake dynamically captures both domain specific nuances and cross-domain patterns, enabling the training of a domain agnostic model. Furthermore, a domain-aware sampling strategy optimizes sample acquisition by prioritizing diverse domain coverage. Experimental results across multiple datasets demonstrate that the proposed approach consistently outperforms various baselines. Our results emphasize that human-LLM co-annotation is a highly cost-effective approach that delivers excellent performance. Evaluations across several datasets show that CoALFake consistently outperforms a range of existing baselines, even with minimal human oversight.

CoALFake: Collaborative Active Learning with Human-LLM Co-Annotation for Cross-Domain Fake News Detection

Abstract

The proliferation of fake news across diverse domains highlights critical limitations in current detection systems, which often exhibit narrow domain specificity and poor generalization. Existing cross-domain approaches face two key challenges: (1) reliance on labelled data, which is frequently unavailable and resource intensive to acquire and (2) information loss caused by rigid domain categorization or neglect of domain-specific features. To address these issues, we propose CoALFake, a novel approach for cross-domain fake news detection that integrates Human-Large Language Model (LLM) co-annotation with domain-aware Active Learning (AL). Our method employs LLMs for scalable, low-cost annotation while maintaining human oversight to ensure label reliability. By integrating domain embedding techniques, the CoALFake dynamically captures both domain specific nuances and cross-domain patterns, enabling the training of a domain agnostic model. Furthermore, a domain-aware sampling strategy optimizes sample acquisition by prioritizing diverse domain coverage. Experimental results across multiple datasets demonstrate that the proposed approach consistently outperforms various baselines. Our results emphasize that human-LLM co-annotation is a highly cost-effective approach that delivers excellent performance. Evaluations across several datasets show that CoALFake consistently outperforms a range of existing baselines, even with minimal human oversight.

Paper Structure

This paper contains 31 sections, 24 equations, 5 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Coss-Domain Fake News Detection
  4. Active Learning for Fake News Detection
  5. Methodology
  6. Domain-Aware Sampling
  7. Human-LLM Co-Annotation
  8. LLM Annotation
  9. Label Verification
  10. Human Re-annotation
  11. Domain Embedding
  12. Domain-agnostic Model
  13. Experiments and Results
  14. Experimental Setup
  15. Model Implementation
  16. ...and 16 more sections

Figures (5)

  • Figure 1: Overview of CoALFake. Data point shapes denote different domains, while colours indicate labelling phases: blue (demonstration set), gray (unlabelled), yellow (Human-LLM labelled) and green (post-classification labels).
  • Figure 2: Overall F1-Scores with different hyperparameters: $\lambda_1, \lambda_2, \lambda_3, \lambda_4, \text{and} \lambda_5$.
  • Figure 3: Overall F1-Scores with different hyperparameters: Embedding Dimension (d), Epochs and Batch Size.
  • Figure 4: CoALFake’s performance with different sampling strategies.
  • Figure 5: F1 Score and Labelling Cost vs. Percentage of Human Re-annotation.