Enhancing BERTopic with Intermediate Layer Representations

TL;DR

This work investigates how BERTopic performance changes when embedding representations from different Transformer layers are used, assessing 18 configurations across three datasets with and without stop-word removal. Using topic coherence ($TC$) via $NPMI$ and topic diversity ($TD$), the study finds that intermediate-layer representations can outperform BERTopic's default setup and even traditional baselines like LDA/NMF in several scenarios. Stop-word removal generally improves both $TC$ and $TD$, and pooling strategy markedly influences results, with Max pooling often yielding higher diversity and Mean pooling boosting coherence on some datasets. The findings underscore the importance of embedding design for topic modeling, including dynamic topic modeling, and point to future work on interpretability and automated topic quality assessment.

Abstract

BERTopic is a topic modeling algorithm that leverages transformer-based embeddings to create dense clusters, enabling the estimation of topic structures and the extraction of valuable insights from a corpus of documents. This approach allows users to efficiently process large-scale text data and gain meaningful insights into its structure. While BERTopic is a powerful tool, embedding preparation can vary, including extracting representations from intermediate model layers and applying transformations to these embeddings. In this study, we evaluate 18 different embedding representations and present findings based on experiments conducted on three diverse datasets. To assess the algorithm's performance, we report topic coherence and topic diversity metrics across all experiments. Our results demonstrate that, for each dataset, it is possible to find an embedding configuration that performs better than the default setting of BERTopic. Additionally, we investigate the influence of stop words on different embedding configurations.

Figures (4)

• Figure 1: Visualization of embedding configurations used in our experiments, "Sum" refers to summing the embeddings, while "Concat" indicates concatenating the representations.
• Figure 2: Topic coherence and topic diversity scores on the United Nations dataset (without stop words) across number of topics (10–50, step 10) for the 5 best-performing configurations.
• Figure 3: Visualization of the top eight topics with words representing them for the worst (Embedding Layer with CLS Pooling) and best (Sum All Layers with Max pooling) configurations.
• Figure 4: Frequency of selected topics estimated from Trump Tweets dataset over the years. Topics have been created by utilizing the best configuration in terms of topic coherence (Concat Last Four Layers with Max pooling).