Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Morphosyntactic probing of multilingual BERT models

Judit Acs, Endre Hamerlik, Roy Schwartz, Noah A. Smith, Andras Kornai

TL;DR

An extensive dataset for multilingual probing of morphological information in language models is introduced and it is found that pre-trained Transformer models (mBERT and XLM-RoBERTa) learn features that attain strong performance across these tasks.

Abstract

We introduce an extensive dataset for multilingual probing of morphological information in language models (247 tasks across 42 languages from 10 families), each consisting of a sentence with a target word and a morphological tag as the desired label, derived from the Universal Dependencies treebanks. We find that pre-trained Transformer models (mBERT and XLM-RoBERTa) learn features that attain strong performance across these tasks. We then apply two methods to locate, for each probing task, where the disambiguating information resides in the input. The first is a new perturbation method that masks various parts of context; the second is the classical method of Shapley values. The most intriguing finding that emerges is a strong tendency for the preceding context to hold more information relevant to the prediction than the following context.

Morphosyntactic probing of multilingual BERT models

TL;DR

An extensive dataset for multilingual probing of morphological information in language models is introduced and it is found that pre-trained Transformer models (mBERT and XLM-RoBERTa) learn features that attain strong performance across these tasks.

Abstract

We introduce an extensive dataset for multilingual probing of morphological information in language models (247 tasks across 42 languages from 10 families), each consisting of a sentence with a target word and a morphological tag as the desired label, derived from the Universal Dependencies treebanks. We find that pre-trained Transformer models (mBERT and XLM-RoBERTa) learn features that attain strong performance across these tasks. We then apply two methods to locate, for each probing task, where the disambiguating information resides in the input. The first is a new perturbation method that masks various parts of context; the second is the classical method of Shapley values. The most intriguing finding that emerges is a strong tendency for the preceding context to hold more information relevant to the prediction than the following context.
Paper Structure (50 sections, 3 equations, 26 figures, 8 tables)

This paper contains 50 sections, 3 equations, 26 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work and Background
  3. Contextual Language Models
  4. Multilingual BERT
  5. XLM-RoBERTa
  6. Directionality
  7. Tokenization
  8. Probing
  9. Data
  10. Choice of Languages and Tags
  11. Data Generation
  12. Methods
  13. Baselines
  14. Experimental Setup
  15. Subword Pooling
  16. ...and 35 more sections

Figures (26)

  • Figure 1: Number of tasks by language family.
  • Figure 2: Probing architecture. Input is tokenized into wordpieces and a weighted sum of the mBERT layers taken on the last wordpiece of the target word is used for classification by an MLP. Only the MLP parameters and the layer weights $w_i$ are trained. $\mathbf{x}_i$ is the output vector of the $i$th layer, $w_i$ is the learned layer weight. The example task here is $\langle$English, NOUN, Number$\rangle$.
  • Figure 3: Difference in accuracy between mBERT (left) and chLSTM, and XLM-RoBERTa (right) and chLSTM grouped by language family and morphological category. Grey cells represent missing tasks.
  • Figure 4: Difference in accuracy between mBERT (left) and chLSTM, and XLM-RoBERTa (right) and chLSTM grouped by language family and POS. Grey cells represent missing tasks.
  • Figure 5: Task-by-task difference between the MLMs and chLSTM in Slavic languages. Grey cells represent missing tasks.
  • ...and 21 more figures