Unveiling the Potential of BERTopic for Multilingual Fake News Analysis -- Use Case: Covid-19

TL;DR

This work assesses BERTopic for multilingual fake-news analysis in the Covid-19 domain using two datasets, GermanFakeNCovid and FakeCovid. It systematically tunes embedding choices, $UMAP$/$HDBSCAN$ hyperparameters via $DBCV$ and evaluates topic extraction with six coherence metrics plus contextualized metrics. Key findings show that embedding choice and cluster granularity substantially influence topic interpretability, with German and US topics more coherent under certain configurations while India presents challenges. The study demonstrates BERTopic's practical applicability for multilingual fake-news analytics and emphasizes the need for diverse hyperparameter exploration and objective topic-similarity measures to guide interpretation.

Abstract

Topic modeling is frequently being used for analysing large text corpora such as news articles or social media data. BERTopic, consisting of sentence embedding, dimension reduction, clustering, and topic extraction, is the newest and currently the SOTA topic modeling method. However, current topic modeling methods have room for improvement because, as unsupervised methods, they require careful tuning and selection of hyperparameters, e.g., for dimension reduction and clustering. This paper aims to analyse the technical application of BERTopic in practice. For this purpose, it compares and selects different methods and hyperparameters for each stage of BERTopic through density based clustering validation and six different topic coherence measures. Moreover, it also aims to analyse the results of topic modeling on real world data as a use case. For this purpose, the German fake news dataset (GermanFakeNCovid) on Covid-19 was created by us and in order to experiment with topic modeling in a multilingual (English and German) setting combined with the FakeCovid dataset. With the final results, we were able to determine thematic similarities between the United States and Germany. Whereas, distinguishing the topics of fake news from India proved to be more challenging.

Figures (2)

• Figure 1: Flowchart of the topic modeling process. In bold the methods applied on each step.
• Figure 2: Clustering results of the fake news of Germany, United States and India using UMAP and HDBSCAN (Top and #C)