Dynamic Embedded Topic Models: properties and recommendations based on diverse corpora

TL;DR

This work investigates how implementation choices in DETM influence performance across five diachronic corpora. It uses systematic hyper-parameter sweeps and reports per-word $NLL$ on test data, with $NPMI$ reported in supplemental material. The study finds that several practical adjustments have limited or inconsistent impact, while vocabulary scaling and small-time-window settings yield reliable performance. It also presents a new pip-installable library and outlines future directions toward continuous-time DETM and richer interpretability measures, with implications for humanities research requiring robust diachronic semantic analysis.

Abstract

We measure the effects of several implementation choices for the Dynamic Embedded Topic Model, as applied to five distinct diachronic corpora, with the goal of isolating important decisions for its use and further development. We identify priorities that will maximize utility in applied scholarship, including the practical scalability of vocabulary size to best exploit the strengths of embedded representations, and more flexible modeling of intervals to accommodate the uneven temporal distributions of historical writing. Of similar importance, we find performance is not significantly or consistently affected by several aspects that otherwise limit the model's application or might consume the resources of a grid search.