Search, Examine and Early-Termination: Fake News Detection with Annotation-Free Evidences

TL;DR

The paper tackles fake-news detection with annotation-free evidences retrieved from the web, addressing the inefficiency and quality-control bottlenecks of prior evidence-based methods. It introduces SEE, a three-stage framework (Search, Examine, Early-Termination) that sequentially fuses news with evidence via attention-enabled transformer decoders and decides when to stop reading more evidences using a confidence assessor. A two-stage training regime first optimizes feature extractors and the final classifier, then trains the assessor with soft targets derived from per-step predictions. Empirical results across four datasets, including those with and without pre-processed evidences, show SEE achieving state-of-the-art accuracy and F1, with robustness to evidence quality and order, highlighting its practicality and reduced reliance on manual evidence curation.

Abstract

Pioneer researches recognize evidences as crucial elements in fake news detection apart from patterns. Existing evidence-aware methods either require laborious pre-processing procedures to assure relevant and high-quality evidence data, or incorporate the entire spectrum of available evidences in all news cases, regardless of the quality and quantity of the retrieved data. In this paper, we propose an approach named \textbf{SEE} that retrieves useful information from web-searched annotation-free evidences with an early-termination mechanism. The proposed SEE is constructed by three main phases: \textbf{S}earching online materials using the news as a query and directly using their titles as evidences without any annotating or filtering procedure, sequentially \textbf{E}xamining the news alongside with each piece of evidence via attention mechanisms to produce new hidden states with retrieved information, and allowing \textbf{E}arly-termination within the examining loop by assessing whether there is adequate confidence for producing a correct prediction. We have conducted extensive experiments on datasets with unprocessed evidences, i.e., Weibo21, GossipCop, and pre-processed evidences, namely Snopes and PolitiFact. The experimental results demonstrate that the proposed method outperforms state-of-the-art approaches.

Figures (4)

• Figure 1: Comparison of methodology between a) previous news-evidence joint learning schemes, typically requiring human-participated evidences and utilizing them altogether, and b) the proposed "Search, Examine and Early-termination" (SEE) scheme, which is capable of utilizing web-searched materials as evidences without annonating and sequentially examining the news alongside each piece of evidence with Early-termination. # denotes the serial number.
• Figure 2: Network design of SEE, our fack-checking FND scheme with annotation-free evidences. SEE includes three main stages: 1. Search: online materials are retrieved as evidences without any annotating or filtering. 2. Examine: the decoders sequentially examine the news alongside with each piece of evidence via self- and cross-attention to acquire more comprehensive information. 3. Early-termination: a shared confidence assessor reduces the hidden states in each time-step into confidence scores and determine either to continue examining more evidences or to terminate and predict.
• Figure 3: Illustration of the two training stages. In stage 1 we solely emphasize feature extraction from evidences from different index in the retrieved queue. In stage 2 the assessor transforms the representations in each time-step into confidence scores for early-exiting on top of fixed feature extractors. The inference stage is depicted in Fig. \ref{['fig:intro']}.
• Figure 4: Quantitative analysis of the impact of $\tau$ on the performance and early-termination.