Detection of Conspiracy Theories Beyond Keyword Bias in German-Language Telegram Using Large Language Models

TL;DR

The paper investigates automated detection of conspiracy theories in German-language Telegram messages without keyword filters, comparing supervised fine-tuning (TelConGBERT) with prompt-based large language models (GPT-3.5, GPT-4, Llama 2) across multiple research questions. TelConGBERT achieves a macro-F1 around 0.85 with strong positive-class performance, and GPT-4 in zero-shot settings attains comparable results, especially when given a tailored conspiracy-definition. Zero-shot prompts generally outperform few-shot prompts, though model robustness varies with prompt design and output constraints. The work demonstrates practical applicability for NGOs and civil society monitoring, highlights transferability within intra-domain time shifts, and discusses ethical considerations and future directions for expanding detection to broader datasets and platforms.

Abstract

The automated detection of conspiracy theories online typically relies on supervised learning. However, creating respective training data requires expertise, time and mental resilience, given the often harmful content. Moreover, available datasets are predominantly in English and often keyword-based, introducing a token-level bias into the models. Our work addresses the task of detecting conspiracy theories in German Telegram messages. We compare the performance of supervised fine-tuning approaches using BERT-like models with prompt-based approaches using Llama2, GPT-3.5, and GPT-4 which require little or no additional training data. We use a dataset of $\sim\!\! 4,000$ messages collected during the COVID-19 pandemic, without the use of keyword filters. Our findings demonstrate that both approaches can be leveraged effectively: For supervised fine-tuning, we report an F1 score of $\sim\!\! 0.8$ for the positive class, making our model comparable to recent models trained on keyword-focused English corpora. We demonstrate our model's adaptability to intra-domain temporal shifts, achieving F1 scores of $\sim\!\! 0.7$. Among prompting variants, the best model is GPT-4, achieving an F1 score of $\sim\!\! 0.8$ for the positive class in a zero-shot setting and equipped with a custom conspiracy theory definition.