Topic Analysis with Side Information: A Neural-Augmented LDA Approach

TL;DR

This work introduces nnLDA, a neural-augmented Latent Dirichlet Allocation that generates document-specific Dirichlet priors $α_d$ from side information via a neural network, enabling nonlinear interactions between auxiliary features and topic distributions. A stochastic variational EM algorithm jointly optimizes the neural parameters $γ$ and the topic-word distribution $β$, with theoretical results showing the nnLDA bound is at least as tight as standard LDA. Empirically, nnLDA outperforms LDA and Dirichlet-Multinomial Regression on multiple datasets in topic coherence, perplexity, and downstream tasks such as classification and comment generation, with larger gains on longer documents. The findings demonstrate that blending neural representation learning with probabilistic topic modeling effectively leverages metadata and labels to improve both interpretability and predictive performance in topic modeling settings.

Abstract

Traditional topic models such as Latent Dirichlet Allocation (LDA) have been widely used to uncover latent structures in text corpora, but they often struggle to integrate auxiliary information such as metadata, user attributes, or document labels. These limitations restrict their expressiveness, personalization, and interpretability. To address this, we propose nnLDA, a neural-augmented probabilistic topic model that dynamically incorporates side information through a neural prior mechanism. nnLDA models each document as a mixture of latent topics, where the prior over topic proportions is generated by a neural network conditioned on auxiliary features. This design allows the model to capture complex nonlinear interactions between side information and topic distributions that static Dirichlet priors cannot represent. We develop a stochastic variational Expectation-Maximization algorithm to jointly optimize the neural and probabilistic components. Across multiple benchmark datasets, nnLDA consistently outperforms LDA and Dirichlet-Multinomial Regression in topic coherence, perplexity, and downstream classification. These results highlight the benefits of combining neural representation learning with probabilistic topic modeling in settings where side information is available.

Figures (6)

• Figure 1: PTS dataset
• Figure 2: WIP dataset
• Figure 3: DCL dataset
• Figure 4: PTS F1 score
• Figure 5: WIP F1 score

Theorems & Definitions (2)

• proof
• proof