Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Is Self-Supervised Learning Enough to Fill in the Gap? A Study on Speech Inpainting

Ihab Asaad, Maxime Jacquelin, Olivier Perrotin, Laurent Girin, Thomas Hueber

TL;DR

This work investigates whether a self-supervised speech encoder trained on a masking pretext can perform speech inpainting without fine-tuning, by pairing HuBERT with a neural vocoder. It compares two SSL-based pipelines—pre-trained HuBERT with a trained HiFi-GAN decoder (I_PT) and fine-tuned HuBERT with a frozen HiFi-GAN (I_FT)—against baselines including an ASR-TTS approach, across in-domain and out-of-domain data, single/multi-speaker conditions, and informed/blind inpainting. Objective and perceptual evaluations show that SSL-based methods outperform baselines, reconstructing gaps up to 200 ms and sometimes 400 ms, with FT favored for single-speaker and PT advantageous for multi-speaker contexts. The results demonstrate that SSL pretext tasks transfer effectively to inpainting, enabling robust speech reconstruction without task-specific SSL fine-tuning in many settings, and they highlight the potential of SSL-based inpainting for efficient, adaptable speech restoration. The study also provides a practical framework, code, and demo to facilitate further exploration of SSL-driven speech inpainting and its relation to linguistic predictability.

Abstract

Speech inpainting consists in reconstructing corrupted or missing speech segments using surrounding context, a process that closely resembles the pretext tasks in Self-Supervised Learning (SSL) for speech encoders. This study investigates using SSL-trained speech encoders for inpainting without any additional training beyond the initial pretext task, and simply adding a decoder to generate a waveform. We compare this approach to supervised fine-tuning of speech encoders for a downstream task -- here, inpainting. Practically, we integrate HuBERT as the SSL encoder and HiFi-GAN as the decoder in two configurations: (1) fine-tuning the decoder to align with the frozen pre-trained encoder's output and (2) fine-tuning the encoder for an inpainting task based on a frozen decoder's input. Evaluations are conducted under single- and multi-speaker conditions using in-domain datasets and out-of-domain datasets (including unseen speakers, diverse speaking styles, and noise). Both informed and blind inpainting scenarios are considered, where the position of the corrupted segment is either known or unknown. The proposed SSL-based methods are benchmarked against several baselines, including a text-informed method combining automatic speech recognition with zero-shot text-to-speech synthesis. Performance is assessed using objective metrics and perceptual evaluations. The results demonstrate that both approaches outperform baselines, successfully reconstructing speech segments up to 200 ms, and sometimes up to 400 ms. Notably, fine-tuning the SSL encoder achieves more accurate speech reconstruction in single-speaker settings, while a pre-trained encoder proves more effective for multi-speaker scenarios. This demonstrates that an SSL pretext task can transfer to speech inpainting, enabling successful speech reconstruction with a pre-trained encoder.

Is Self-Supervised Learning Enough to Fill in the Gap? A Study on Speech Inpainting

TL;DR

This work investigates whether a self-supervised speech encoder trained on a masking pretext can perform speech inpainting without fine-tuning, by pairing HuBERT with a neural vocoder. It compares two SSL-based pipelines—pre-trained HuBERT with a trained HiFi-GAN decoder (I_PT) and fine-tuned HuBERT with a frozen HiFi-GAN (I_FT)—against baselines including an ASR-TTS approach, across in-domain and out-of-domain data, single/multi-speaker conditions, and informed/blind inpainting. Objective and perceptual evaluations show that SSL-based methods outperform baselines, reconstructing gaps up to 200 ms and sometimes 400 ms, with FT favored for single-speaker and PT advantageous for multi-speaker contexts. The results demonstrate that SSL pretext tasks transfer effectively to inpainting, enabling robust speech reconstruction without task-specific SSL fine-tuning in many settings, and they highlight the potential of SSL-based inpainting for efficient, adaptable speech restoration. The study also provides a practical framework, code, and demo to facilitate further exploration of SSL-driven speech inpainting and its relation to linguistic predictability.

Abstract

Speech inpainting consists in reconstructing corrupted or missing speech segments using surrounding context, a process that closely resembles the pretext tasks in Self-Supervised Learning (SSL) for speech encoders. This study investigates using SSL-trained speech encoders for inpainting without any additional training beyond the initial pretext task, and simply adding a decoder to generate a waveform. We compare this approach to supervised fine-tuning of speech encoders for a downstream task -- here, inpainting. Practically, we integrate HuBERT as the SSL encoder and HiFi-GAN as the decoder in two configurations: (1) fine-tuning the decoder to align with the frozen pre-trained encoder's output and (2) fine-tuning the encoder for an inpainting task based on a frozen decoder's input. Evaluations are conducted under single- and multi-speaker conditions using in-domain datasets and out-of-domain datasets (including unseen speakers, diverse speaking styles, and noise). Both informed and blind inpainting scenarios are considered, where the position of the corrupted segment is either known or unknown. The proposed SSL-based methods are benchmarked against several baselines, including a text-informed method combining automatic speech recognition with zero-shot text-to-speech synthesis. Performance is assessed using objective metrics and perceptual evaluations. The results demonstrate that both approaches outperform baselines, successfully reconstructing speech segments up to 200 ms, and sometimes up to 400 ms. Notably, fine-tuning the SSL encoder achieves more accurate speech reconstruction in single-speaker settings, while a pre-trained encoder proves more effective for multi-speaker scenarios. This demonstrates that an SSL pretext task can transfer to speech inpainting, enabling successful speech reconstruction with a pre-trained encoder.
Paper Structure (51 sections, 13 equations, 5 figures, 3 tables)

This paper contains 51 sections, 13 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Speech inpainting
  4. Early signal processing approaches
  5. Deep learning based approaches
  6. Multimodal speech inpainting
  7. Speech enhancement in noise, speech restoration, and speech coding
  8. Speech continuation with speech language models
  9. Conclusion
  10. Method
  11. Problem formulation
  12. Masking, the core pretext task of HuBERT
  13. Linking masking, prediction and inpainting
  14. Combining HuBERT encoder with HiFi-GAN decoder
  15. Pre-trained SSL
  16. ...and 36 more sections

Figures (5)

  • Figure 1: Proposed inpainting framework, leveraging a pre-trained SSL encoder on a unmasking task and using a neural vocoder for audio generation. Top: the SSL encoder is kept frozen (pre-trained) while the neural vocoder is adapted to the SSL output. Bottom: the SSL encoder is fine-tuned to an inpainting task, while the neural vocoder is kept frozen. In the present study we use HuBERT as the SSL and HiFi-GAN as the vocoder (subfigure in the middle). The inpainting process and adaptation mechanism between the SSL output and neural vocoder input are detailed in \ref{['section:method']}).
  • Figure 2: Illustration of how the HuBERT model works during training (top-left) and inference (top-right). Adaptation of HuBERT for inference within the proposed inpainting pipeline, in the informed (middle) and blind (bottom) inpainting configurations.
  • Figure 3: ASR-TTS baseline $\mathcal{I}_{\mathtt{ASR}}$ combining automatic speech recognition (ASR) and zero-shot text-to-speech (TTS) for text-informed speech inpainting. The corresponding inpainting process is detailed in \ref{['subsec-baselines']}).
  • Figure 4: Examples of inpainted speech signals (80-dimensional Mel-spectrograms, informed case). Left: single-speaker for the sentence "no su[gge]stion was made", Right: multi-speaker for the sentence "(...) has do[ne a goo]d job". The green rectangles illustrate the position and length of the mask (200ms).
  • Figure 5: Boxplots illustrating the distribution of the MUSHRA scores for the two models $\mathcal{I}_{\mathtt{FT}}$ and $\mathcal{I}_{\mathtt{PT}}$, and for the $\mathcal{I}_{\mathtt{LI}}$ and $\mathcal{I}_{\mathtt{ASR}}$ baselines (informed inpainting with a 200ms-length mask). *** indicates that the differences between each pairs of inpainting frameworks were found very significant (i.e. $p \leq 0.001$).