Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Prediction

TL;DR

MR-HuBERT introduces a hierarchical, multi-resolution self-supervised speech model that jointly learns high- and low-resolution representations via a two-stream Transformer architecture and multi-resolution masked unit prediction. By integrating sampling-based up/downscaling and HuBERT-style targets, the approach achieves substantial gains on LibriSpeech, SUPERB, and ML-SUPERB while delivering 9–13% faster inference through reduced sequence lengths. Across ASR, multilingual, and task-averaged benchmarks, MR-HuBERT consistently surpasses or matches HuBERT baselines, with notable benefits in low-resource and multilingual settings. The work demonstrates the practicality of multi-resolution pre-training for speech and provides open-source resources to facilitate broader adoption and further study.

Abstract

Existing Self-Supervised Learning (SSL) models for speech typically process speech signals at a fixed resolution of 20 milliseconds. This approach overlooks the varying informational content present at different resolutions in speech signals. In contrast, this paper aims to incorporate multi-resolution information into speech self-supervised representation learning. We introduce a SSL model that leverages a hierarchical Transformer architecture, complemented by HuBERT-style masked prediction objectives, to process speech at multiple resolutions. Experimental results indicate that the proposed model not only achieves more efficient inference but also exhibits superior or comparable performance to the original HuBERT model over various tasks. Specifically, significant performance improvements over the original HuBERT have been observed in fine-tuning experiments on the LibriSpeech speech recognition benchmark as well as in evaluations using the Speech Universal PERformance Benchmark (SUPERB) and Multilingual SUPERB (ML-SUPERB).

Figures (4)

• Figure 1: MR-HuBERT pre-training framework. The framework utilizes multi-resolution masked units prediction. The details of each module are discussed in Section \ref{['sec: method']}
• Figure 2: Sampling modules. The proposed sampling modules utilize a residual-based learning framework in either upsampling or downsampling. Details of the module are discussed in Section \ref{['ssec: sampling module']}.
• Figure 3: Layer-weight analysis on SUPERB tasks over two base models. The weights are the layer-wise weights after the Softmax function, which are trained together with downstream models as detailed in Section \ref{['ssec: superb evaluation']}.
• Figure 4: Layer-weight analysis on SUPERB tasks over two large models. The weights are the layer-wise weights after the Softmax function, which are trained together with downstream models as detailed in Section \ref{['ssec: superb evaluation']}.