Evaluating Negative Sampling Approaches for Neural Topic Models

TL;DR

This work investigates decoder-side negative sampling in neural topic models within a unified OCTIS-based framework, evaluating seven VAE-based approaches across four public datasets. By combining reconstruction and regularization losses with contrastive or triplet-based signals, the decoder-negative-sampling models consistently improve topic coherence (NPMI, CV) and maintain topic diversity (IRBO), while also enhancing downstream document classification accuracy. The study provides a comprehensive taxonomy, quantitative and qualitative analyses, latent-space visualizations, and robustness checks (vocabulary size and training time), demonstrating practical benefits and outlining directions for theoretical grounding. Overall, decoder-focused negative sampling emerges as a robust and scalable technique to boost neural topic modeling performance, with clear guidance for practitioners and a reproducible framework for future work.

Abstract

Negative sampling has emerged as an effective technique that enables deep learning models to learn better representations by introducing the paradigm of learn-to-compare. The goal of this approach is to add robustness to deep learning models to learn better representation by comparing the positive samples against the negative ones. Despite its numerous demonstrations in various areas of computer vision and natural language processing, a comprehensive study of the effect of negative sampling in an unsupervised domain like topic modeling has not been well explored. In this paper, we present a comprehensive analysis of the impact of different negative sampling strategies on neural topic models. We compare the performance of several popular neural topic models by incorporating a negative sampling technique in the decoder of variational autoencoder-based neural topic models. Experiments on four publicly available datasets demonstrate that integrating negative sampling into topic models results in significant enhancements across multiple aspects, including improved topic coherence, richer topic diversity, and more accurate document classification. Manual evaluations also indicate that the inclusion of negative sampling into neural topic models enhances the quality of the generated topics. These findings highlight the potential of negative sampling as a valuable tool for advancing the effectiveness of neural topic models.

Figures (7)

• Figure 1: Topic model categories based on negative sampling techniques.
• Figure 2: A general framework for VAE-based topic models.
• Figure 3: A general framework for the VAE-based topic models that uses negative sampling methodology on the encoder.
• Figure 4: A general framework for the VAE-based topic models that uses negative sampling methodology on the decoder.
• Figure 5: The variation of topic coherence (NPMI and CV) and topic diversity (IRBO) with topic count are shown for different topic models on four datasets. The ordinate value of each data point reports the median over five independent runs.