Multi-environment Topic Models

TL;DR

This work introduces the Multi-environment Topic Model (MTM), an unsupervised probabilistic model that separates global and environment-specific terms and shows that the MTM produces interpretable global topics with distinct environment-specific words.

Abstract

Probabilistic topic models are a powerful tool for extracting latent themes from large text datasets. In many text datasets, we also observe per-document covariates (e.g., source, style, political affiliation) that act as environments that modulate a "global" (environment-agnostic) topic representation. Accurately learning these representations is important for prediction on new documents in unseen environments and for estimating the causal effect of topics on real-world outcomes. To this end, we introduce the Multi-environment Topic Model (MTM), an unsupervised probabilistic model that separates global and environment-specific terms. Through experimentation on various political content, from ads to tweets and speeches, we show that the MTM produces interpretable global topics with distinct environment-specific words. On multi-environment data, the MTM outperforms strong baselines in and out-of-distribution. It also enables the discovery of accurate causal effects.

Figures (3)

• Figure 1: A graphical model for the multi-environment topic model (MTM). $M$ denotes words in a document and $D$ documents. $E$ denotes the environments documents are drawn from (determined by different configurations of covariates $\mathbf{x}$). $z$ denotes topic assignment, $\beta$ denotes global weights for each word in the vocabulary, and $\gamma$ denotes environment-specific weights.
• Figure 2: Perplexity on held-out data across models trained on the ideological dataset, consisting of political advertisements from Republican and Democrat politicians. The MTM $+ \gamma_R$ represents global $\beta$ with Republican-specific deviations $\gamma_R$. MTM outperforms all baselines on all three test sets.
• Figure 3: Perplexity on held-out data across models trained on a dataset of political advertisements from channels across different regions of the U.S. The MTM $+ \gamma_R$ represents global $\beta$ with Republican-specific deviations $\gamma_R$. MTM outperforms all baselines across all regions.