Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LinguAlchemy: Fusing Typological and Geographical Elements for Unseen Language Generalization

Muhammad Farid Adilazuarda, Samuel Cahyawijaya, Alham Fikri Aji, Genta Indra Winata, Ayu Purwarianti

TL;DR

The authors' LinguAlchemy significantly improves the performance of mBERT and XLM-R on low-resource languages in multiple downstream tasks such as intent classification, news classification, and semantic relatedness compared to fully finetuned models and displaying a high degree of unseen language generalization.

Abstract

Pretrained language models (PLMs) have become remarkably adept at task and language generalization. Nonetheless, they often fail when faced with unseen languages. In this work, we present LinguAlchemy, a regularization method that incorporates various linguistic information covering typological, geographical, and phylogenetic features to align PLMs representation to the corresponding linguistic information on each language. Our LinguAlchemy significantly improves the performance of mBERT and XLM-R on low-resource languages in multiple downstream tasks such as intent classification, news classification, and semantic relatedness compared to fully finetuned models and displaying a high degree of unseen language generalization. We further introduce AlchemyScale and AlchemyTune, extension of LinguAlchemy which adjusts the linguistic regularization weights automatically, alleviating the need for hyperparameter search.

LinguAlchemy: Fusing Typological and Geographical Elements for Unseen Language Generalization

TL;DR

The authors' LinguAlchemy significantly improves the performance of mBERT and XLM-R on low-resource languages in multiple downstream tasks such as intent classification, news classification, and semantic relatedness compared to fully finetuned models and displaying a high degree of unseen language generalization.

Abstract

Pretrained language models (PLMs) have become remarkably adept at task and language generalization. Nonetheless, they often fail when faced with unseen languages. In this work, we present LinguAlchemy, a regularization method that incorporates various linguistic information covering typological, geographical, and phylogenetic features to align PLMs representation to the corresponding linguistic information on each language. Our LinguAlchemy significantly improves the performance of mBERT and XLM-R on low-resource languages in multiple downstream tasks such as intent classification, news classification, and semantic relatedness compared to fully finetuned models and displaying a high degree of unseen language generalization. We further introduce AlchemyScale and AlchemyTune, extension of LinguAlchemy which adjusts the linguistic regularization weights automatically, alleviating the need for hyperparameter search.
Paper Structure (24 sections, 2 equations, 4 figures, 12 tables, 1 algorithm)

This paper contains 24 sections, 2 equations, 4 figures, 12 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Unseen Languages Adaptation with LinguAlchemy
  4. Does Multilingual LMs capture Linguistic Constraints?
  5. Syntax Features
  6. Geographical Features
  7. Proof of Concept
  8. Linguistic Separability in LMs
  9. $\vcenter{}$LinguAlchemy
  10. Dynamic Scaling Approaches
  11. Experiment Setting
  12. Datasets
  13. Models
  14. Results and Discussion
  15. LinguAlchemy Performance
  16. ...and 9 more sections

Figures (4)

  • Figure 1: LinguAlchemy enhances performance in unseen languages by allowing the model to predict the linguistic vector and then fitting it via a similarity loss towards the specific language's URIEL vector.
  • Figure 2: Alignment between mBERT Representation with URIEL Language Representation. The green-shaded areas indicate the sentence representations of mBERT while the brown dots represent the URIEL representations of the corresponding language.
  • Figure 3: Average performance of unseen languages under various URIEL loss scaling factors.
  • Figure 4: Model performance across language families. Dotted lines indicates language families used in training in some of the training stages (solid dots for active use--refer to Table \ref{['tab:language_groups']}), and solid grey lines for families unseen in all training stages, with variance shown in shading.