Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GrEmLIn: A Repository of Green Baseline Embeddings for 87 Low-Resource Languages Injected with Multilingual Graph Knowledge

Daniil Gurgurov, Rishu Kumar, Simon Ostermann

TL;DR

GrEmLIn tackles the scarcity of high-quality static embeddings for 87 low- and mid-resource languages by enriching GloVe with multilingual graph knowledge from ConceptNet. The method merges GloVe with graph-derived PPMI signals via a truncated SVD and a subsequent linear projection, producing 300-dimensional embeddings for the full vocabulary. Empirically, graph-enhanced GloVe outperforms vanilla GloVe, FastText, and, in many cases, contextual models on lexical similarity and several downstream tasks (sentiment, NLI), while remaining a lightweight, parameter-free-at-inference baseline. This work demonstrates the enduring value of static, graph-informed embeddings as a scalable, environmentally friendly alternative for multilingual NLP in data-scarce settings, and makes the GrEmLIn resources publicly available on HuggingFace. The results underscore the importance of integrating structured multilingual knowledge to bolster static representations in low-resource languages, with room for further fusion and coverage improvements in future work.

Abstract

Contextualized embeddings based on large language models (LLMs) are available for various languages, but their coverage is often limited for lower resourced languages. Using LLMs for such languages is often difficult due to a high computational cost; not only during training, but also during inference. Static word embeddings are much more resource-efficient ("green"), and thus still provide value, particularly for very low-resource languages. There is, however, a notable lack of comprehensive repositories with such embeddings for diverse languages. To address this gap, we present GrEmLIn, a centralized repository of green, static baseline embeddings for 87 mid- and low-resource languages. We compute GrEmLIn embeddings with a novel method that enhances GloVe embeddings by integrating multilingual graph knowledge, which makes our static embeddings competitive with LLM representations, while being parameter-free at inference time. Our experiments demonstrate that GrEmLIn embeddings outperform state-of-the-art contextualized embeddings from E5 on the task of lexical similarity. They remain competitive in extrinsic evaluation tasks like sentiment analysis and natural language inference, with average performance gaps of just 5-10\% or less compared to state-of-the-art models, given a sufficient vocabulary overlap with the target task, and underperform only on topic classification. Our code and embeddings are publicly available at https://huggingface.co/DFKI.

GrEmLIn: A Repository of Green Baseline Embeddings for 87 Low-Resource Languages Injected with Multilingual Graph Knowledge

TL;DR

GrEmLIn tackles the scarcity of high-quality static embeddings for 87 low- and mid-resource languages by enriching GloVe with multilingual graph knowledge from ConceptNet. The method merges GloVe with graph-derived PPMI signals via a truncated SVD and a subsequent linear projection, producing 300-dimensional embeddings for the full vocabulary. Empirically, graph-enhanced GloVe outperforms vanilla GloVe, FastText, and, in many cases, contextual models on lexical similarity and several downstream tasks (sentiment, NLI), while remaining a lightweight, parameter-free-at-inference baseline. This work demonstrates the enduring value of static, graph-informed embeddings as a scalable, environmentally friendly alternative for multilingual NLP in data-scarce settings, and makes the GrEmLIn resources publicly available on HuggingFace. The results underscore the importance of integrating structured multilingual knowledge to bolster static representations in low-resource languages, with room for further fusion and coverage improvements in future work.

Abstract

Contextualized embeddings based on large language models (LLMs) are available for various languages, but their coverage is often limited for lower resourced languages. Using LLMs for such languages is often difficult due to a high computational cost; not only during training, but also during inference. Static word embeddings are much more resource-efficient ("green"), and thus still provide value, particularly for very low-resource languages. There is, however, a notable lack of comprehensive repositories with such embeddings for diverse languages. To address this gap, we present GrEmLIn, a centralized repository of green, static baseline embeddings for 87 mid- and low-resource languages. We compute GrEmLIn embeddings with a novel method that enhances GloVe embeddings by integrating multilingual graph knowledge, which makes our static embeddings competitive with LLM representations, while being parameter-free at inference time. Our experiments demonstrate that GrEmLIn embeddings outperform state-of-the-art contextualized embeddings from E5 on the task of lexical similarity. They remain competitive in extrinsic evaluation tasks like sentiment analysis and natural language inference, with average performance gaps of just 5-10\% or less compared to state-of-the-art models, given a sufficient vocabulary overlap with the target task, and underperform only on topic classification. Our code and embeddings are publicly available at https://huggingface.co/DFKI.
Paper Structure (27 sections, 2 equations, 1 figure, 9 tables)

This paper contains 27 sections, 2 equations, 1 figure, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Graph knowledge sources.
  4. Word embeddings.
  5. Improving Embeddings with Knowledge Graphs.
  6. Method
  7. GloVe Embeddings
  8. Graph Embeddings
  9. Singular Value Decomposition (SVD)
  10. Linear Transformation
  11. Experiments
  12. Languages
  13. Evaluation Data
  14. Experimental Setup
  15. Results
  16. ...and 12 more sections

Figures (1)

  • Figure 1: Scatter plots illustrating the relationship between vocabulary overlap and performance improvements across various language tasks using GloVe and graph-enhanced embeddings (G+P). Each plot shows the improvement in performance (G+P - GloVe) versus the common vocabulary size (log-scaled). Solid lines represent the best-fit log-linear trend.