Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DQ-Data2vec: Decoupling Quantization for Multilingual Speech Recognition

Qijie Shao, Linhao Dong, Kun Wei, Sining Sun, Lei Xie

TL;DR

DQ-Data2vec tackles the coupling of language and phoneme information in Data2vec by introducing decoupling quantization: two online K-means quantizers operate on targeted teacher-layer representations to produce explicit language- and phoneme-related targets. The method supports both shallow (unsupervised) and deep (weakly supervised) decoupling, achieving substantial improvements on CommonVoice: relative PER reductions of $9.51\%$ (self-supervised) and $18.09\%$ (weakly-supervised), and relative WER reductions of $11.58\%$ and $1.55\%$, respectively. Across extensive ablations, the authors show that explicit layer selection, pooling, and group quantization are critical for stable training and clustering quality, with online K-means outperforming Gumbel-softmax in this setting. The results demonstrate that decoupled language and phoneme representations enable more effective multilingual ASR with a single-stage pre-training regime and offer a path toward applying quantized codebooks to larger language models.

Abstract

Data2vec is a self-supervised learning (SSL) approach that employs a teacher-student architecture for contextual representation learning via masked prediction, demonstrating remarkable performance in monolingual ASR. Previous studies have revealed that data2vec's shallow layers capture speaker and language information, middle layers encode phoneme and word features, while deep layers are responsible for reconstruction. Language and phoneme features are crucial for multilingual ASR. However, data2vec's masked representation generation relies on multi-layer averaging, inevitably coupling these features. To address this limitation, we propose a decoupling quantization based data2vec (DQ-Data2vec) for multilingual ASR, which includes a data2vec backbone and two improved online K-means quantizers. Our core idea is using the K-means quantizer with specified cluster numbers to decouple language and phoneme information for masked prediction. Specifically, in the language quantization, considering that the number of languages is significantly different from other irrelevant features (e.g., speakers), we assign the cluster number to match the number of languages, explicitly decoupling shallow layers' language-related information from irrelevant features. This strategy is also applied to decoupling middle layers' phoneme and word features. In a self-supervised scenario, experiments on the CommonVoice dataset demonstrate that DQ-Data2vec achieves a relative reduction of 9.51% in phoneme error rate (PER) and 11.58% in word error rate (WER) compared to data2vec and UniData2vec. Moreover, in a weakly-supervised scenario incorporating language labels and high-resource language text labels, the relative reduction is 18.09% and 1.55%, respectively.

DQ-Data2vec: Decoupling Quantization for Multilingual Speech Recognition

TL;DR

DQ-Data2vec tackles the coupling of language and phoneme information in Data2vec by introducing decoupling quantization: two online K-means quantizers operate on targeted teacher-layer representations to produce explicit language- and phoneme-related targets. The method supports both shallow (unsupervised) and deep (weakly supervised) decoupling, achieving substantial improvements on CommonVoice: relative PER reductions of (self-supervised) and (weakly-supervised), and relative WER reductions of and , respectively. Across extensive ablations, the authors show that explicit layer selection, pooling, and group quantization are critical for stable training and clustering quality, with online K-means outperforming Gumbel-softmax in this setting. The results demonstrate that decoupled language and phoneme representations enable more effective multilingual ASR with a single-stage pre-training regime and offer a path toward applying quantized codebooks to larger language models.

Abstract

Data2vec is a self-supervised learning (SSL) approach that employs a teacher-student architecture for contextual representation learning via masked prediction, demonstrating remarkable performance in monolingual ASR. Previous studies have revealed that data2vec's shallow layers capture speaker and language information, middle layers encode phoneme and word features, while deep layers are responsible for reconstruction. Language and phoneme features are crucial for multilingual ASR. However, data2vec's masked representation generation relies on multi-layer averaging, inevitably coupling these features. To address this limitation, we propose a decoupling quantization based data2vec (DQ-Data2vec) for multilingual ASR, which includes a data2vec backbone and two improved online K-means quantizers. Our core idea is using the K-means quantizer with specified cluster numbers to decouple language and phoneme information for masked prediction. Specifically, in the language quantization, considering that the number of languages is significantly different from other irrelevant features (e.g., speakers), we assign the cluster number to match the number of languages, explicitly decoupling shallow layers' language-related information from irrelevant features. This strategy is also applied to decoupling middle layers' phoneme and word features. In a self-supervised scenario, experiments on the CommonVoice dataset demonstrate that DQ-Data2vec achieves a relative reduction of 9.51% in phoneme error rate (PER) and 11.58% in word error rate (WER) compared to data2vec and UniData2vec. Moreover, in a weakly-supervised scenario incorporating language labels and high-resource language text labels, the relative reduction is 18.09% and 1.55%, respectively.
Paper Structure (18 sections, 12 equations, 3 figures, 6 tables)

This paper contains 18 sections, 12 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Self-supervised Speech Representation Learning
  4. Weakly-supervised Speech Representation Learning
  5. Decoupling Specific Target for SSL
  6. Method
  7. Teacher-Student Backbone
  8. Shallow Decoupling
  9. Deep Decoupling
  10. Experiments Setup
  11. Dataset
  12. Implementation Details
  13. Experiments Results
  14. Shallow Decoupling
  15. Deep Decoupling
  16. ...and 3 more sections

Figures (3)

  • Figure 1: The framework overview of our proposed DQ-Data2vec. The symbol ${\ast}$ denotes that the object is only employed in the deep decoupling scenario. In the right figure, the gray dashed lines represent different designs for frame-level and utterance-level quantization, while the red and blue dashed lines represent the gradient back-propagation paths of different loss functions.
  • Figure 2: The conditional probability ${P(lang|code)}$ on CommonVoice test set. The y-axis is the language set sorted by the number of occurrences, and the x-axis is the active codewords sorted by the most correlated language. In the figure, ACN denotes the active codeword number, while AGN indicates the active clusters within the two clustering groups. LP signifies language purity, and LNMI refers to language-normalized mutual information.
  • Figure 3: The conditional probability ${P(phone|code)}$ on CommonVoice test set. The y-axis is the phoneme set sorted by the number of occurrences, and the x-axis is the active codewords sorted by the most correlated phoneme. In the figure, ACN denotes the active codeword number, while AGN indicates the active clusters within the two clustering groups. PP signifies phoneme purity, and PNMI refers to phoneme-normalized mutual information.