Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discovery

Angelo Ortiz Tandazo, Manel Khentout, Youssef Benchekroun, Thomas Hueber, Emmanuel Dupoux

TL;DR

MauBERT extends HuBERT with articulatory-feature supervision across 55 languages to induce universal phonetic biases for robust cross-lingual representations. By pre-training on a multilingual VoxCommunis corpus and applying self-supervised fine-tuning with linguistically informed targets, the model achieves strong ABX-based invariance and effective zero-shot and few-shot adaptation to unseen languages. The work demonstrates two training paradigms (AF-feature and phone-prediction) and analyzes their impact on phonetic discrimination, enabling phonetic inventory discovery in low-resource languages. The findings highlight the potential of linguistically informed supervision to make multilingual speech technologies more inclusive while outlining practical directions for future improvements and domain adaptation.

Abstract

This paper introduces MauBERT, a multilingual extension of HuBERT that leverages articulatory features for robust cross-lingual phonetic representation learning. We continue HuBERT pre-training with supervision based on a phonetic-to-articulatory feature mapping in 55 languages. Our models learn from multilingual data to predict articulatory features or phones, resulting in language-independent representations that capture multilingual phonetic properties. Through comprehensive ABX discriminability testing, we show MauBERT models produce more context-invariant representations than state-of-the-art multilingual self-supervised learning models. Additionally, the models effectively adapt to unseen languages and casual speech with minimal self-supervised fine-tuning (10 hours of speech). This establishes an effective approach for instilling linguistic inductive biases in self-supervised speech models.

MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discovery

TL;DR

MauBERT extends HuBERT with articulatory-feature supervision across 55 languages to induce universal phonetic biases for robust cross-lingual representations. By pre-training on a multilingual VoxCommunis corpus and applying self-supervised fine-tuning with linguistically informed targets, the model achieves strong ABX-based invariance and effective zero-shot and few-shot adaptation to unseen languages. The work demonstrates two training paradigms (AF-feature and phone-prediction) and analyzes their impact on phonetic discrimination, enabling phonetic inventory discovery in low-resource languages. The findings highlight the potential of linguistically informed supervision to make multilingual speech technologies more inclusive while outlining practical directions for future improvements and domain adaptation.

Abstract

This paper introduces MauBERT, a multilingual extension of HuBERT that leverages articulatory features for robust cross-lingual phonetic representation learning. We continue HuBERT pre-training with supervision based on a phonetic-to-articulatory feature mapping in 55 languages. Our models learn from multilingual data to predict articulatory features or phones, resulting in language-independent representations that capture multilingual phonetic properties. Through comprehensive ABX discriminability testing, we show MauBERT models produce more context-invariant representations than state-of-the-art multilingual self-supervised learning models. Additionally, the models effectively adapt to unseen languages and casual speech with minimal self-supervised fine-tuning (10 hours of speech). This establishes an effective approach for instilling linguistic inductive biases in self-supervised speech models.
Paper Structure (46 sections, 2 figures, 8 tables)

This paper contains 46 sections, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multilingual Speech Representation Learning.
  4. Articulatory Features in Speech Processing.
  5. Evaluation of Speech Representations.
  6. MauBERT
  7. Multilingual Pre-Training
  8. Encoder.
  9. Downstream modules.
  10. Phone model.
  11. Self-Supervised Fine-Tuning
  12. MauBERT Multilingual Pre-Training
  13. Data Processing
  14. Training Details
  15. Evaluation and Results
  16. ...and 31 more sections

Figures (2)

  • Figure 1: a. Multilingual training.MauBERT-feat is trained to recognise ternary-valued articulatory features and phones using an encoder (HuBERT-base), downstream modules (weighted sum, up-projection, two-layer BLSTM, feature projection), and a phone model (two-layer perceptron); the feature states receive no gradients from the phone recognition loss due to the stop-gradient operator (sg). b. Self-supervised fine-tuning.Top: Offline clustering is applied to one of the layers of MauBERT, the teacher network, on an unseen language; bottom: the MauBERT Transformer, the student network, is then trained to predict the corresponding clusters of masked input.
  • Figure 2: Reduction of the triphone ABX error rates across the 5.0 development languages and 5.0 test languages between the base HuBERT model, and MauBERT, tested in zero-shot and after masked fine-tuning (10) with or without labels on a new language. The two speaker conditions are averaged, and the 10 subset is chosen for the test languages. $^*$Mandarin and Wolof only have [list-units = single]1.5;1.8 of training data, resp.