Quantifying consistency and accuracy of Latent Dirichlet Allocation

TL;DR

This work addresses the problem of instability in probabilistic topic models like LDA, where repeated runs yield different latent topics due to stochasticity. It introduces a stability distance that fuses accuracy and reproducibility by generating ground-truth corpora from the LDA process and evaluating outputs across 50 repeats with varying topic counts, using Jensen-Shannon, Jaccard, and Rank-Biased Overlap measures. The findings show LDA is internally consistent (producing similar topics across runs) but not necessarily accurate to the ground truth, yet it can correctly identify the true number of topics in controlled simulations and on 20 Newsgroups data; stability improves when topics are more separable. The proposed stability framework is general and applicable to other topic models, offering a practical tool for assessing stability and guiding model choice in topic analysis, while acknowledging limitations of bag-of-words generation and suggesting future work with syntactic structure via HMMs or LLMs.

Abstract

Topic modelling in Natural Language Processing uncovers hidden topics in large, unlabelled text datasets. It is widely applied in fields such as information retrieval, content summarisation, and trend analysis across various disciplines. However, probabilistic topic models can produce different results when rerun due to their stochastic nature, leading to inconsistencies in latent topics. Factors like corpus shuffling, rare text removal, and document elimination contribute to these variations. This instability affects replicability, reliability, and interpretation, raising concerns about whether topic models capture meaningful topics or just noise. To address these problems, we defined a new stability measure that incorporates accuracy and consistency and uses the generative properties of LDA to generate a new corpus with ground truth. These generated corpora are run through LDA 50 times to determine the variability in the output. We show that LDA can correctly determine the underlying number of topics in the documents. We also find that LDA is more internally consistent, as the multiple reruns return similar topics; however, these topics are not the true topics.

Figures (3)

• Figure 1: The plot shows the stability measure as the number of topics increases, where $K = 10$ is the true number of topics. The different colours indicate different distributions used in generation, and each plot represents the different similarity measures. For most of the distributions, we see that it dips or plateaus around $K=10$, indicating the measure is most stable at that point. It can also be seen that when there is a lack of signal in the corpus, looking at $N_{10}$, the measurements do not perform well.
• Figure 2: Between: comparing 'truth' to output, within: comparing generated values to each other. The x-axis represents different distributions used in generation. The different markers are between comparisons and within comparisons. We see that for most measures, the LDA output performs worse when compared to the true number of distributions.
• Figure 3: The stability distance measure applied 20Newsgroup dataset over a range of the number of topics. We see that the measures dip around $K=20$, which is the true number of topics.