On Self-improving Token Embeddings
Mario M. Kubek, Shiraj Pokharel, Thomas Böhme, Emma L. McDaniel, Herwig Unger, Armin R. Mikler
TL;DR
The paper tackles the problem of domain-specific word representations by proposing a fast, self-improving method that iteratively refines pre-trained embeddings using contextual information within a fixed window. The core approach updates a target token's vector with $e_{t_{new}} = e_{t_{current}} + \alpha \sum_{n \\in c} e_{n_{current}}$, followed by normalization $e_{t_{new}} = \frac{e_{t_{new}}}{\| e_{t_{new}} \|}$, and stores embedding history to capture topical evolution, with optional imputation for missing embeddings. Demonstrations on general-domain GoogleNewsVectors and NOAA storm narratives show improved topic alignment and effective handling of out-of-vocabulary terms, while remaining computationally lightweight and not reliant on large neural models. The approach supports corpus exploration, conceptual search, and adaptive, production-ready data pipelines, offering practical benefits for real-time, domain-specific NLP tasks and potential extensions to symbolic reasoning and multilingual deployment.
Abstract
This article introduces a novel and fast method for refining pre-trained static word or, more generally, token embeddings. By incorporating the embeddings of neighboring tokens in text corpora, it continuously updates the representation of each token, including those without pre-assigned embeddings. This approach effectively addresses the out-of-vocabulary problem, too. Operating independently of large language models and shallow neural networks, it enables versatile applications such as corpus exploration, conceptual search, and word sense disambiguation. The method is designed to enhance token representations within topically homogeneous corpora, where the vocabulary is restricted to a specific domain, resulting in more meaningful embeddings compared to general-purpose pre-trained vectors. As an example, the methodology is applied to explore storm events and their impacts on infrastructure and communities using narratives from a subset of the NOAA Storm Events database. The article also demonstrates how the approach improves the representation of storm-related terms over time, providing valuable insights into the evolving nature of disaster narratives.