Speech Representation Learning Revisited: The Necessity of Separate Learnable Parameters and Robust Data Augmentation

TL;DR

The paper tackles the problem that speech representations encoding both content and other information compete for model capacity, leading to suboptimal downstream performance. It introduces O-HuBERT, which uses a separate learnable pathway (via a USP token) to model 'other' information while preserving content modeling through HuBERT's masked prediction objective. A two-stage data augmentation strategy and a composite loss combining $L_{MPL}$, $L_{USP}$, and regularization terms yield state-of-the-art results on the SUPERB benchmark with a ~100M-parameter model pre-trained on ~960 hours, supporting the JOOCI (Joint Optimization of Other and Content Information) vision. The results highlight the importance of dedicated mechanisms for orthogonal information types in speech pre-training and point to future work on multiple USP tokens and joint optimization approaches that avoid trade-offs between information channels.

Abstract

Speech modeling methods learn one embedding for a fixed segment of speech, typically in between 10-25 ms. The information present in speech can be divided into two categories: "what is being said" (content) and "how it is expressed" (other) and these two are orthogonal in nature causing the optimization algorithm to find a sub-optimal solution if forced to optimize together. This leads to sub-optimal performance in one or all downstream tasks as shown by previous studies. Current self-supervised learning (SSL) methods such as HuBERT are very good at modeling the content information present in speech. Data augmentation improves the performance on tasks which require effective modeling of other information but this leads to a divided capacity of the model. In this work, we conduct a preliminary study to understand the importance of modeling other information using separate learnable parameters. We propose a modified version of HuBERT, termed Other HuBERT (O-HuBERT), to test our hypothesis. Our findings are twofold: first, the O-HuBERT method is able to utilize all layers to build complex features to encode other information; second, a robust data augmentation strategy is essential for learning the information required by tasks that depend on other information and to achieve state-of-the-art (SOTA) performance on the SUPERB benchmark with a similarly sized model (100 million parameters) and pre-training data (960 hours).

Figures (3)

• Figure 1: The figure illustrates the processing of two speech utterances in a batch. In our proposed O-HuBERT model, content information is encoded using masked token indices, while other information is modeled using the USP token. Notably, calculating the other information loss involves pairs of samples from the batch, whereas the content loss does not require such pairs.
• Figure 2: Importance of each transformer encoder layer (x-axis) vs different tasks for three configurations denoted as (i) G (other information only), (ii) L (content information only), and (iii) GL (combined other and content information).
• Figure 3: Comparison of the GL and L setup dev and test accuracy plots on the SID task from the SUPERB benchmark.