Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Geographic Adaptation of Pretrained Language Models

Valentin Hofmann, Goran Glavaš, Nikola Ljubešić, Janet B. Pierrehumbert, Hinrich Schütze

TL;DR

This work addresses the lack of extralinguistic, geographic grounding in pretrained language models by introducing geoadaptation, an intermediate multi-task training step that jointly optimizes masked language modeling with token-level geolocation regression. The authors apply this approach to four PLMs across three geographic regions (AGS, BCMS, DNS) and a combined EUR setting, achieving strong improvements across five downstream tasks, particularly in zero-shot scenarios, and attaining new state-of-the-art results for geolocation and language identification. They demonstrate that geoadaptation reshapes the PLM latent space into a geographically retrofit topology, enabling robust generalization without supervision from toponym names during task fine-tuning. The results highlight the potential for grounding NLP systems in geography to enhance performance on geolinguistic tasks and suggest avenues for extending the method to autoregressive models and broader extralinguistic knowledge integration.

Abstract

While pretrained language models (PLMs) have been shown to possess a plethora of linguistic knowledge, the existing body of research has largely neglected extralinguistic knowledge, which is generally difficult to obtain by pretraining on text alone. Here, we contribute to closing this gap by examining geolinguistic knowledge, i.e., knowledge about geographic variation in language. We introduce geoadaptation, an intermediate training step that couples language modeling with geolocation prediction in a multi-task learning setup. We geoadapt four PLMs, covering language groups from three geographic areas, and evaluate them on five different tasks: fine-tuned (i.e., supervised) geolocation prediction, zero-shot (i.e., unsupervised) geolocation prediction, fine-tuned language identification, zero-shot language identification, and zero-shot prediction of dialect features. Geoadaptation is very successful at injecting geolinguistic knowledge into the PLMs: the geoadapted PLMs consistently outperform PLMs adapted using only language modeling (by especially wide margins on zero-shot prediction tasks), and we obtain new state-of-the-art results on two benchmarks for geolocation prediction and language identification. Furthermore, we show that the effectiveness of geoadaptation stems from its ability to geographically retrofit the representation space of the PLMs.

Geographic Adaptation of Pretrained Language Models

TL;DR

This work addresses the lack of extralinguistic, geographic grounding in pretrained language models by introducing geoadaptation, an intermediate multi-task training step that jointly optimizes masked language modeling with token-level geolocation regression. The authors apply this approach to four PLMs across three geographic regions (AGS, BCMS, DNS) and a combined EUR setting, achieving strong improvements across five downstream tasks, particularly in zero-shot scenarios, and attaining new state-of-the-art results for geolocation and language identification. They demonstrate that geoadaptation reshapes the PLM latent space into a geographically retrofit topology, enabling robust generalization without supervision from toponym names during task fine-tuning. The results highlight the potential for grounding NLP systems in geography to enhance performance on geolinguistic tasks and suggest avenues for extending the method to autoregressive models and broader extralinguistic knowledge integration.

Abstract

While pretrained language models (PLMs) have been shown to possess a plethora of linguistic knowledge, the existing body of research has largely neglected extralinguistic knowledge, which is generally difficult to obtain by pretraining on text alone. Here, we contribute to closing this gap by examining geolinguistic knowledge, i.e., knowledge about geographic variation in language. We introduce geoadaptation, an intermediate training step that couples language modeling with geolocation prediction in a multi-task learning setup. We geoadapt four PLMs, covering language groups from three geographic areas, and evaluate them on five different tasks: fine-tuned (i.e., supervised) geolocation prediction, zero-shot (i.e., unsupervised) geolocation prediction, fine-tuned language identification, zero-shot language identification, and zero-shot prediction of dialect features. Geoadaptation is very successful at injecting geolinguistic knowledge into the PLMs: the geoadapted PLMs consistently outperform PLMs adapted using only language modeling (by especially wide margins on zero-shot prediction tasks), and we obtain new state-of-the-art results on two benchmarks for geolocation prediction and language identification. Furthermore, we show that the effectiveness of geoadaptation stems from its ability to geographically retrofit the representation space of the PLMs.
Paper Structure (6 sections, 2 equations, 6 figures, 8 tables)

This paper contains 6 sections, 2 equations, 6 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Geoadaptation
  4. Experimental Setup
  5. Results and Analysis
  6. Conclusion

Figures (6)

  • Figure 1: Geographic distribution of the data for AGS (left), BCMS (middle), and DNS (right). Each point represents a Jodel post (AGS) or tweet (BCMS, DNS). Point density correlates with population density, with the densest areas corresponding to urban centers. For DNS, we exclude the Svalbard islands, which do not have any points.
  • Figure 2: Confusion matrices for MLMAda (a), GeoAda-S (b), and GeoAda-W (c) on ZS-Geoloc (BCMS). While MLMAda always predicts one of the three most frequent city tokens (Beograd, Sarajevo, or Zagreb), the predictions of GeoAda-S and GeoAda-W are much more diverse and less tied to frequency.
  • Figure 3: Performance on BCMS ZS-Geoloc (a), ZS-Lang (b), and ZS-Dialect (c, d) for different number of epochs. In stark contrast to geoadaptation (GeoAda-S, GeoAda-W), language modeling adaptation alone (MLMAda) barely helps in acquiring geographic knowledge (a), which is also reflected by the consistently worse performance on ZS-Lang (b). MLMAda does form dialectal associations after several epochs, but the inductive bias of geoadaptation allows GeoAda-S and GeoAda-W to establish those associations more quickly (c, d).
  • Figure 4: (Geo-)adaptation diagnostics. The figure illustrates how log perplexity of language modeling (a) and median distance of token-level geolocation prediction (b) change on dev during BCMS geoadaptation.
  • Figure 5: Association strength between the BERTić embeddings of Croatian/Serbian cities and ije/e variants for MLMAda (top) and GeoAda-W (bottom), measured using WEAT Caliskan.2017. A positive or negative score indicates that a city is associated more strongly with the ije or e variants, respectively.
  • ...and 1 more figures