Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Speech Representations with Variational Predictive Coding

Sung-Lin Yeh, Peter Bell, Hao Tang

TL;DR

The paper provides a principled variational predictive coding interpretation of the HuBERT objective, framing speech representation learning as learning latent codes that predict masked or quantized frames. It derives an objective that unifies reconstruction and predictive terms, and demonstrates extensions such as soft assignments and joint optimization that yield tangible improvements. Empirically, better pre-training under this framework improves phone classification, f0 tracking, speaker recognition, and ASR, and the framework reveals meaningful connections to APC, CPC, wav2vec 2.0, WavLM, and BEST-RQ. Overall, the work grounds self-supervised speech objectives in predictive coding theory and offers practical avenues for parameterization, optimization, and task impact.

Abstract

Despite being the best known objective for learning speech representations, the HuBERT objective has not been further developed and improved. We argue that it is the lack of an underlying principle that stalls the development, and, in this paper, we show that predictive coding under a variational view is the principle behind the HuBERT objective. Due to its generality, our formulation provides opportunities to improve parameterization and optimization, and we show two simple modifications that bring immediate improvements to the HuBERT objective. In addition, the predictive coding formulation has tight connections to various other objectives, such as APC, CPC, wav2vec, and BEST-RQ. Empirically, the improvement in pre-training brings significant improvements to four downstream tasks: phone classification, f0 tracking, speaker recognition, and automatic speech recognition, highlighting the importance of the predictive coding interpretation.

Learning Speech Representations with Variational Predictive Coding

TL;DR

The paper provides a principled variational predictive coding interpretation of the HuBERT objective, framing speech representation learning as learning latent codes that predict masked or quantized frames. It derives an objective that unifies reconstruction and predictive terms, and demonstrates extensions such as soft assignments and joint optimization that yield tangible improvements. Empirically, better pre-training under this framework improves phone classification, f0 tracking, speaker recognition, and ASR, and the framework reveals meaningful connections to APC, CPC, wav2vec 2.0, WavLM, and BEST-RQ. Overall, the work grounds self-supervised speech objectives in predictive coding theory and offers practical avenues for parameterization, optimization, and task impact.

Abstract

Despite being the best known objective for learning speech representations, the HuBERT objective has not been further developed and improved. We argue that it is the lack of an underlying principle that stalls the development, and, in this paper, we show that predictive coding under a variational view is the principle behind the HuBERT objective. Due to its generality, our formulation provides opportunities to improve parameterization and optimization, and we show two simple modifications that bring immediate improvements to the HuBERT objective. In addition, the predictive coding formulation has tight connections to various other objectives, such as APC, CPC, wav2vec, and BEST-RQ. Empirically, the improvement in pre-training brings significant improvements to four downstream tasks: phone classification, f0 tracking, speaker recognition, and automatic speech recognition, highlighting the importance of the predictive coding interpretation.
Paper Structure (38 sections, 18 equations, 2 figures, 10 tables)

This paper contains 38 sections, 18 equations, 2 figures, 10 tables.

Table of Contents

  1. Introduction
  2. A Variational Framework for Predictive Coding
  3. HuBERT as Predictive Coding
  4. Masked Prediction
  5. Quantization and Reconstruction
  6. Predicting Quantized Frames
  7. Two-Step Optimization
  8. Extensions
  9. Softening the Point Mass
  10. Approximating the Expectation with Sampling
  11. Future Prediction
  12. Connections to Prior Work
  13. APC and VQ-APC
  14. MPC and DeCoAR
  15. HuBERT, WavLM, and BEST-RQ
  16. ...and 23 more sections

Figures (2)

  • Figure 1: HuBERT as variational predictive coding. The set $v_1, \dots, v_K$ are the codewords in the codebook, $x_{\backslash M}$ are the unmasked frames, and $x_M$ are the masked frames. There are two loss functions involved, the Kullback-Leibler divergence ($\mathrm{KL}$) and the mean-squared error (MSE).
  • Figure 2: The training losses of different optimization approaches implemented by our own Base (top) and in fairseq Base (bottom). The final loss values can be found in Table \ref{['tab:ctc']}.