Towards Real-Time Fake News Detection under Evidence Scarcity

TL;DR

This work tackles real-time fake news detection under evidence scarcity by introducing EASE, a sequential framework that adaptively selects among three perspectives—evidence-based, reasoning-based, and sentiment-based fallback—guided by evaluators trained with instruction tuning and pseudo labels. The Evidence Evaluator and Evidence Expert form the core of the evidence-based path, while aReasoning Evaluator/Expert and a Sentiment Expert handle cases when external evidence is insufficient. A new benchmark, RealTimeNews-25, assesses performance on recent, scarce-evidence events, and extensive experiments show state-of-the-art results on historical data and strong generalization to real-time news. The approach emphasizes interpretability and evaluation-aware decision-making, enabling robust fake news detection in time-sensitive settings with limited external proof.

Abstract

Fake news detection becomes particularly challenging in real-time scenarios, where emerging events often lack sufficient supporting evidence. Existing approaches often rely heavily on external evidence and therefore struggle to generalize under evidence scarcity. To address this issue, we propose Evaluation-Aware Selection of Experts (EASE), a novel framework for real-time fake news detection that dynamically adapts its decision-making process according to the assessed sufficiency of available evidence. EASE introduces a sequential evaluation mechanism comprising three independent perspectives: (1) Evidence-based evaluation, which assesses evidence and incorporates it into decision-making only when the evidence is sufficiently supportive; (2) Reasoning-based evaluation, which leverages the world knowledge of large language models (LLMs) and applies them only when their reliability is adequately established; and (3) Sentiment-based fallback, which integrates sentiment cues when neither evidence nor reasoning is reliable. To enhance the accuracy of evaluation processes, EASE employs instruction tuning with pseudo labels to guide each evaluator in justifying its perspective-specific knowledge through interpretable reasoning. Furthermore, the expert modules integrate the evaluators' justified assessments with the news content to enable evaluation-aware decision-making, thereby enhancing overall detection accuracy. Moreover, we introduce RealTimeNews-25, a new benchmark comprising recent news for evaluating model generalization on emerging news with limited evidence. Extensive experiments demonstrate that EASE not only achieves state-of-the-art performance across multiple benchmarks, but also significantly improves generalization to real-time news. The code and dataset are available: https://github.com/wgyhhhh/EASE.

Figures (4)

• Figure 1: Examples of limited supporting evidence: (a) conflicting information across different sources, and (b) the absence of relevant evidence.
• Figure 2: The Evaluation-Aware Selection of Experts (EASE) framework. EASE first employs an evidence-based strategy, where (1) the evidence agent retrieves external evidence, (2) the evaluator assesses its sufficiency, and (3) the evidence expert makes the final decision. When the evidence is insufficient, EASE switches to a reasoning-based strategy that mirrors the evidence-based pipeline, generating and evaluating reasoning knowledge for decision-making. If the reasoning knowledge is unreliable, EASE activates a sentiment-based strategy, which analyzes stylistic and emotional cues to produce the final judgment.
• Figure 3: (a) Activation ratios of different experts in EASE across datasets. (b) Evaluation accuracy of evaluators. represent the evidence evaluator before and after fine-tuning, respectively, while represent the reasoning evaluator before and after fine-tuning, respectively.
• Figure 4: Prompting process of the evidence agent.