An Embedded Diachronic Sense Change Model with a Case Study from Ancient Greek

TL;DR

EDiSC, an Embedded DiSC model, which combines word embeddings with DiSC to provide superior model performance is introduced, and it is shown empirically that EDiSC offers improved predictive accuracy, ground-truth recovery and uncertainty quantification, as well as better sampling efficiency and scalability properties with MCMC methods.

Abstract

Word meanings change over time, and word senses evolve, emerge or die out in the process. For ancient languages, where the corpora are often small and sparse, modelling such changes accurately proves challenging, and quantifying uncertainty in sense-change estimates consequently becomes important. GASC (Genre-Aware Semantic Change) and DiSC (Diachronic Sense Change) are existing generative models that have been used to analyse sense change for target words from an ancient Greek text corpus, using unsupervised learning without the help of any pre-training. These models represent the senses of a given target word such as "kosmos" (meaning decoration, order or world) as distributions over context words, and sense prevalence as a distribution over senses. The models are fitted using Markov Chain Monte Carlo (MCMC) methods to measure temporal changes in these representations. This paper introduces EDiSC, an Embedded DiSC model, which combines word embeddings with DiSC to provide superior model performance. It is shown empirically that EDiSC offers improved predictive accuracy, ground-truth recovery and uncertainty quantification, as well as better sampling efficiency and scalability properties with MCMC methods. The challenges of fitting these models are also discussed.

Figures (9)

• Figure 1: EDiSC plate diagram for three time periods. Dashed nodes are constant parameters, solid black nodes are latent variables and solid red nodes are observed variables. $D_{g,t}$ is the number of snippets for genre $g$ at time $t$.
• Figure 2: WAIC and Brier scores for different choices of embedding dimension $M$ for the "bank", "kosmos" and "mus" data
• Figure 3: "Kosmos" expert-annotated empirical sense prevalence (coloured bars with height $N_{k,g,t}^{o} / \sum_{l=1'}^{K'} N_{l,g,t}^{o}$ for each $k,g,t$), and 95% HPD intervals (error bars) and posterior means (circles) from the model output. Snippet counts $N_{\cdot,g,t}^{o}$ are given in brackets below the axes. Note that the labelled posteriors $\tilde{\phi}|z$ from DiSC and EDiSC are identical.
• Figure 4: Mean run times in CPU seconds for 500 MCMC iterations on synthetic data using different models, vocabulary sizes ($V$) and number of snippets ($D)$
• Figure 5: "Mus" expert-annotated empirical sense prevalence (coloured bars), and 95% HPD intervals (error bars) and posterior means (circles) from the model output