Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Designing and Contextualising Probes for African Languages

Wisdom Aduah, Francois Meyer

TL;DR

This work provides the first systematic probing of linguistic knowledge in PLMs for African languages, analyzing six typologically diverse languages across POS tagging, NER, and news topic classification. It introduces a MasakhaPOS control task to disentangle model knowledge from probe memorisation and evaluates seven PLMs with layer-wise probes, revealing that token-level syntax concentrates in middle-to-deep layers while sentence-level semantics distribute across layers. Multilingual adaptation and language inclusion in pretraining consistently improve probe performance, with AfroXLMR-large and Nguni-XLMR-large often attaining the strongest results, though transfer benefits vary by language representation. The study offers actionable insights for interpretability and model design in African-language NLP, highlighting both the promise of cross-lingual transfer and the remaining gaps for underrepresented languages like Igbo.

Abstract

Pretrained language models (PLMs) for African languages are continually improving, but the reasons behind these advances remain unclear. This paper presents the first systematic investigation into probing PLMs for linguistic knowledge about African languages. We train layer-wise probes for six typologically diverse African languages to analyse how linguistic features are distributed. We also design control tasks, a way to interpret probe performance, for the MasakhaPOS dataset. We find PLMs adapted for African languages to encode more linguistic information about target languages than massively multilingual PLMs. Our results reaffirm previous findings that token-level syntactic information concentrates in middle-to-last layers, while sentence-level semantic information is distributed across all layers. Through control tasks and probing baselines, we confirm that performance reflects the internal knowledge of PLMs rather than probe memorisation. Our study applies established interpretability techniques to African-language PLMs. In doing so, we highlight the internal mechanisms underlying the success of strategies like active learning and multilingual adaptation.

Designing and Contextualising Probes for African Languages

TL;DR

This work provides the first systematic probing of linguistic knowledge in PLMs for African languages, analyzing six typologically diverse languages across POS tagging, NER, and news topic classification. It introduces a MasakhaPOS control task to disentangle model knowledge from probe memorisation and evaluates seven PLMs with layer-wise probes, revealing that token-level syntax concentrates in middle-to-deep layers while sentence-level semantics distribute across layers. Multilingual adaptation and language inclusion in pretraining consistently improve probe performance, with AfroXLMR-large and Nguni-XLMR-large often attaining the strongest results, though transfer benefits vary by language representation. The study offers actionable insights for interpretability and model design in African-language NLP, highlighting both the promise of cross-lingual transfer and the remaining gaps for underrepresented languages like Igbo.

Abstract

Pretrained language models (PLMs) for African languages are continually improving, but the reasons behind these advances remain unclear. This paper presents the first systematic investigation into probing PLMs for linguistic knowledge about African languages. We train layer-wise probes for six typologically diverse African languages to analyse how linguistic features are distributed. We also design control tasks, a way to interpret probe performance, for the MasakhaPOS dataset. We find PLMs adapted for African languages to encode more linguistic information about target languages than massively multilingual PLMs. Our results reaffirm previous findings that token-level syntactic information concentrates in middle-to-last layers, while sentence-level semantic information is distributed across all layers. Through control tasks and probing baselines, we confirm that performance reflects the internal knowledge of PLMs rather than probe memorisation. Our study applies established interpretability techniques to African-language PLMs. In doing so, we highlight the internal mechanisms underlying the success of strategies like active learning and multilingual adaptation.
Paper Structure (20 sections, 1 equation, 7 figures, 4 tables)

This paper contains 20 sections, 1 equation, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. PLMs for African Languages
  4. Contextualising Probe Performance
  5. Probing Framework
  6. Probe Design
  7. MasakhaPOS Control Task
  8. NTC and NER Probe Baselines
  9. Experimental Setup
  10. Data
  11. Languages
  12. PLMs
  13. Results
  14. POS Tagging
  15. NER
  16. ...and 5 more sections

Figures (7)

  • Figure 1: POS probe performance (selectivity), averaged over 6 African languages.
  • Figure 2: NER probe gains (over random baselines) across layers, averaged over 6 African languages.
  • Figure 3: Probe selectivity for POS tagging (the difference between MasakhaPOS accuracy and control task accuracy), across all layers and 6 African languages.
  • Figure 4: Probe performance gains for NER tagging (F1 improvements over randomly re-initialised PLM architectures), across all layers and 6 African languages.
  • Figure 5: Probe accuracy for news topic classification (visualised in comparison to a random contextual baseline) across all layers and 6 African languages.
  • ...and 2 more figures