Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MSR-HuBERT: Self-supervised Pre-training for Adaptation to Multiple Sampling Rates

Zikang Huang, Meng Ge, Tianrui Wang, Xuanchen Li, Xiaobao Wang, Longbiao Wang, Jianwu Dang

Abstract

Self-supervised learning (SSL) has advanced speech processing. However, existing speech SSL methods typically assume a single sampling rate and struggle with mixed-rate data due to temporal resolution mismatch. To address this limitation, we propose MSRHuBERT, a multi-sampling-rate adaptive pre-training method. Building on HuBERT, we replace its single-rate downsampling CNN with a multi-sampling-rate adaptive downsampling CNN that maps raw waveforms from different sampling rates to a shared temporal resolution without resampling. This design enables unified mixed-rate pre-training and fine-tuning. In experiments spanning 16 to 48 kHz, MSRHuBERT outperforms HuBERT on speech recognition and full-band speech reconstruction, preserving high-frequency detail while modeling low-frequency semantic structure. Moreover, MSRHuBERT retains HuBERT's mask-prediction objective and Transformer encoder, so existing analyses and improvements that were developed for HuBERT can apply directly.

MSR-HuBERT: Self-supervised Pre-training for Adaptation to Multiple Sampling Rates

Abstract

Self-supervised learning (SSL) has advanced speech processing. However, existing speech SSL methods typically assume a single sampling rate and struggle with mixed-rate data due to temporal resolution mismatch. To address this limitation, we propose MSRHuBERT, a multi-sampling-rate adaptive pre-training method. Building on HuBERT, we replace its single-rate downsampling CNN with a multi-sampling-rate adaptive downsampling CNN that maps raw waveforms from different sampling rates to a shared temporal resolution without resampling. This design enables unified mixed-rate pre-training and fine-tuning. In experiments spanning 16 to 48 kHz, MSRHuBERT outperforms HuBERT on speech recognition and full-band speech reconstruction, preserving high-frequency detail while modeling low-frequency semantic structure. Moreover, MSRHuBERT retains HuBERT's mask-prediction objective and Transformer encoder, so existing analyses and improvements that were developed for HuBERT can apply directly.
Paper Structure (11 sections, 4 equations, 3 figures, 3 tables)

This paper contains 11 sections, 4 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. HuBERT
  3. Proposed Method
  4. Multi‑sampling‑rate Adaptive Downsampling CNN
  5. Mixed-rate Mask Prediction via Single Codebook
  6. Experiments
  7. Pre-training Setup
  8. Fine-tuning Setup
  9. Evaluation on SR and ASR
  10. Exploration for Learning Paradigm
  11. Conclusion

Figures (3)

  • Figure 1: The resolution mismatch across diverse sampling rates with a fixed 320× downsampling CNN.
  • Figure 2: The architecture of MSRHuBERT, which takes raw waveforms at different sampling rates as input. The multi-sampling-rate adaptive downsampling CNN can map waveforms to a common temporal resolution by designing rate-specific downsampling, and supports mixed-rate pre-training.
  • Figure 3: Weight analysis on the ASR task of the SUPERB Benchmark. Layer 0 corresponds to the input of the first Transformer layer. The y-axis represents different settings, including pre-trained model and sampling rate of the downstream dataset in ASR.