Rumour Evaluation with Very Large Language Models

TL;DR

This paper investigates leveraging prompting-based Very Large Language Models (GPT-3.5-turbo and GPT-4) to combat misinformation on Twitter by extending the RumourEval tasks of veracity prediction and SDQC stance classification. It systematically explores zero-, one-, and few-shot prompting for veracity (with 2-way, 3-way, and 2-step schemes) and zero-shot prompting for stance (3-way, 4-way, multiclass), delivering predictions with confidence scores and natural-language justifications. The authors achieve substantial improvements over 2017 RumourEval baselines for veracity, particularly with zero-shot GPT-4 in binary classification, while stance results are mixed, with multiclass extending the original SDQC framework but not consistently beating baselines. The work contributes a comprehensive prompt-engineering study, provides open-source code and experimental settings, and highlights the potential and limitations of VLLMs for AI-for-social-good in misinformation mitigation. Overall, the results underscore the promise of prompt-driven explanations and trust signals from VLLMs, while pointing to the need for broader datasets, multilingual evaluation, and model diversity for robust social media misinformation analysis.

Abstract

Conversational prompt-engineering-based large language models (LLMs) have enabled targeted control over the output creation, enhancing versatility, adaptability and adhoc retrieval. From another perspective, digital misinformation has reached alarming levels. The anonymity, availability and reach of social media offer fertile ground for rumours to propagate. This work proposes to leverage the advancement of prompting-dependent LLMs to combat misinformation by extending the research efforts of the RumourEval task on its Twitter dataset. To the end, we employ two prompting-based LLM variants (GPT-3.5-turbo and GPT-4) to extend the two RumourEval subtasks: (1) veracity prediction, and (2) stance classification. For veracity prediction, three classifications schemes are experimented per GPT variant. Each scheme is tested in zero-, one- and few-shot settings. Our best results outperform the precedent ones by a substantial margin. For stance classification, prompting-based-approaches show comparable performance to prior results, with no improvement over finetuning methods. Rumour stance subtask is also extended beyond the original setting to allow multiclass classification. All of the generated predictions for both subtasks are equipped with confidence scores determining their trustworthiness degree according to the LLM, and post-hoc justifications for explainability and interpretability purposes. Our primary aim is AI for social good.