FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Model

TL;DR

FASTopic is proposed, a fast, adaptive, stable, and transferable topic model that follows a new paradigm: Dual Semantic-relation Reconstruction (DSR), and shows superior effectiveness, efficiency, adaptivity, stability, and transferability, compared to state-of-the-art baselines across various scenarios.

Abstract

Topic models have been evolving rapidly over the years, from conventional to recent neural models. However, existing topic models generally struggle with either effectiveness, efficiency, or stability, highly impeding their practical applications. In this paper, we propose FASTopic, a fast, adaptive, stable, and transferable topic model. FASTopic follows a new paradigm: Dual Semantic-relation Reconstruction (DSR). Instead of previous conventional, VAE-based, or clustering-based methods, DSR directly models the semantic relations among document embeddings from a pretrained Transformer and learnable topic and word embeddings. By reconstructing through these semantic relations, DSR discovers latent topics. This brings about a neat and efficient topic modeling framework. We further propose a novel Embedding Transport Plan (ETP) method. Rather than early straightforward approaches, ETP explicitly regularizes the semantic relations as optimal transport plans. This addresses the relation bias issue and thus leads to effective topic modeling. Extensive experiments on benchmark datasets demonstrate that our FASTopic shows superior effectiveness, efficiency, adaptivity, stability, and transferability, compared to state-of-the-art baselines across various scenarios.

Figures (8)

• Figure 1: (a): Running speed rank and overall performance rank on the experiments with 6 benchmark datasets, including topic quality, doc-topic distribution quality, downstream tasks, and transferability. (b): Running time under the WoS dataset with varying sizes. See complete results in \ref{['fig_training_time_dataset_size_full']}.
• Figure 2: Illustration of topic modeling paradigms. (a): VAE-based topic modeling with an encoder and a decoder zhao2021neuralwang2022representingwu2023effective. (b): Clustering-based topic modeling by clustering document embeddings angelov2020top2vecgrootendorst2022bertopic. (c): Dual Semantic-relation Reconstruction (DSR), modeling doc-topic distributions as the semantic relations between document ($\blacksquare$) and topic embeddings ($\blacktriangle$), and modeling topic-word distributions as the semantic relations between topic ($\blacktriangle$) and word embeddings ($\bullet$). Here we model these relations as the transport plans to alleviate the relation bias issue.
• Figure 3: (a, b): Relation weights of topics to documents. (c, d): t-SNE visualization Maaten2008 of document ($\blacksquare$), and topic ($\blacktriangle$) embeddings under 50 topics ($K\!\!=\!\!50$). While most topic embeddings gather together in Parameterized Softmax (a,c) as it causes biased relations, ETP (b,d) separates all topic embeddings with regularized relations, avoiding the bias issue.
• Figure 4: (Left): Text classification results of Accuracy (Acc) and F1. (Right): Transferability results. We use topic models trained on Wikitext-103 to infer the doc-topic distributions of other datasets. The best is in rgb] .886, .937, .855bold.
• Figure 5: A code example of using FASTopic. Install FASTopic via pip and use its APIs to handle a dataset.