Stick-Breaking Embedded Topic Model with Continuous Optimal Transport for Online Analysis of Document Streams

TL;DR

This paper tackles online topic modeling for document streams by removing the need to fix the number of topics and by preserving the geometric integrity of topic embeddings over time. It introduces SB-SETM, which combines a truncated stick-breaking prior (replacing the logistic–normal prior) with a continuous optimal transport-based merging mechanism that transports current topic embeddings into the latent space of the previous timestep. Theoretical contributions include a reparameterizable variational family using Kumaraswamy sticks and a low-rank Monge map for high-dimensional OT, guarded by a balanced ELBO with reconstruction and regularization terms. Empirically, SB-SETM demonstrates improved topic activation accuracy and coherent evolution on both synthetic data and real-world corpora, including 20kNewsGroupOnline and UkRuWarNews22-23, with evidence that COT-based merging yields better coherence, diversity, and temporal alignment than DOT-based alternatives. The work advances online topic modeling by enabling scalable, adaptive topic discovery and stable topic trajectories in evolving document streams, with practical implications for real-time analysis of news and other streaming sources.

Abstract

Online topic models are unsupervised algorithms to identify latent topics in data streams that continuously evolve over time. Although these methods naturally align with real-world scenarios, they have received considerably less attention from the community compared to their offline counterparts, due to specific additional challenges. To tackle these issues, we present SB-SETM, an innovative model extending the Embedded Topic Model (ETM) to process data streams by merging models formed on successive partial document batches. To this end, SB-SETM (i) leverages a truncated stick-breaking construction for the topic-per-document distribution, enabling the model to automatically infer from the data the appropriate number of active topics at each timestep; and (ii) introduces a merging strategy for topic embeddings based on a continuous formulation of optimal transport adapted to the high dimensionality of the latent topic space. Numerical experiments show SB-SETM outperforming baselines on simulated scenarios. We extensively test it on a real-world corpus of news articles covering the Russian-Ukrainian war throughout 2022-2023.

Figures (10)

• Figure 1: The proposed SB-SETM. At each time step, SB-ETM$_t$ receives a new corpus of documents and infers the topic embeddings. These embeddings are then projected into the latent space of the previous timestep via the transport map $\underset{t\rightarrow t-1}{{T^\star} }$.
• Figure 2: Merging of topic embeddings between $t-1$ and $t$. Latent space of $\alpha^{(t-1)}$ in blue and of $\Tilde{\alpha}^{(t)}$ in orange. First, the topic embeddings at time $t$ are transported using the Monge map $\underset{t\rightarrow t-1}{{T^\star} }$ between two high-dimensional Gaussian distributions; then topics that are not in the $\varepsilon$-neighborhood of an existing topic will be considered as new topics (denoted by a star).
• Figure 3: Distribution of $K_\texttt{pred}$ (log-scale) at each timestep across 8 training runs at different $K_{\texttt{init}}$. The solid lines connect the median values of the corresponding box plots.
• Figure 4: PCA projections of topic embeddings (at timesteps $t=1$–$2$ for (a)-(b), and $t=11$–$12$ for (c)-(d)). Arrows indicate associations between topics across consecutive timesteps. In (c), the topics in blue are: god/jesus/interpretation/moral/belief, space/spacecraft/field/orbit/earth, dealer/engine/driver/rear/ford.
• Figure 5: Topic-term count matrix on UkRuWarNews22-23, SB-SETM ($K_{\texttt{init}} = 50$). Each row is a predicted global topic, and each column is a dataset category. The entries indicate how many documents of a given class are assigned to each topic, rescaled with inverse column weights to account for class imbalance.