Evaluating the Performance of Large Language Models in Scientific Claim Detection and Classification

TL;DR

This work evaluates multiple large language models (LLMs) on detecting and classifying COVID-19–related scientific claims in tweets, addressing the Digital Infodemic with automated fact-checking. Using a specialized COVID-19 dataset (covid19dataset) and a framework of system prompts, few-shot examples, and instructional patterns, the study compares Llama 2 variants and GPT-3.5/4 across two tasks: claim existence and claim verifiability, reporting practical performance gaps and strengths. GPT-4 consistently yields the strongest results, while recall remains a challenge across models, indicating missed verifiable claims and possible annotation or linguistic nuances in tweets. The paper highlights the pivotal role of prompts, discusses practical deployment considerations (GPU/cloud resources), and points to Retrieval Augmented Generation (RAG) and fine-tuning as promising avenues to enhance automated misinformation mitigation for public health communication.

Abstract

The pervasive influence of social media during the COVID-19 pandemic has been a double-edged sword, enhancing communication while simultaneously propagating misinformation. This \textit{Digital Infodemic} has highlighted the urgent need for automated tools capable of discerning and disseminating factual content. This study evaluates the efficacy of Large Language Models (LLMs) as innovative solutions for mitigating misinformation on platforms like Twitter. LLMs, such as OpenAI's GPT and Meta's LLaMA, offer a pre-trained, adaptable approach that bypasses the extensive training and overfitting issues associated with traditional machine learning models. We assess the performance of LLMs in detecting and classifying COVID-19-related scientific claims, thus facilitating informed decision-making. Our findings indicate that LLMs have significant potential as automated fact-checking tools, though research in this domain is nascent and further exploration is required. We present a comparative analysis of LLMs' performance using a specialized dataset and propose a framework for their application in public health communication.