Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Open Implementation and Study of BEST-RQ for Speech Processing

Ryan Whetten, Titouan Parcollet, Marco Dinarelli, Yannick Estève

TL;DR

A re-implementation of a Random-projection quantizer is described and it is shown that a random projection quantizer can achieve similar downstream performance as wav2vec 2.0 while decreasing training time by over a factor of two.

Abstract

Self-Supervised Learning (SSL) has proven to be useful in various speech tasks. However, these methods are generally very demanding in terms of data, memory, and computational resources. BERT-based Speech pre-Training with Random-projection Quantizer (BEST-RQ), is an SSL method that has shown great performance on Automatic Speech Recognition (ASR) while being simpler than other SSL methods, such as wav2vec 2.0. Despite BEST-RQ's great performance, details are lacking in the original paper, such as the amount of GPU/TPU hours used in pre-training, and there is no official easy-to-use open-source implementation. Furthermore, BEST-RQ has not been evaluated on other downstream tasks aside from ASR and speech translation. In this work, we describe a re-implementation of a Random-projection quantizer and perform a preliminary study with a comparison to wav2vec 2.0 on four downstream tasks. We discuss the details and differences of our implementation. We show that a random projection quantizer can achieve similar downstream performance as wav2vec 2.0 while decreasing training time by over a factor of two.

Open Implementation and Study of BEST-RQ for Speech Processing

TL;DR

A re-implementation of a Random-projection quantizer is described and it is shown that a random projection quantizer can achieve similar downstream performance as wav2vec 2.0 while decreasing training time by over a factor of two.

Abstract

Self-Supervised Learning (SSL) has proven to be useful in various speech tasks. However, these methods are generally very demanding in terms of data, memory, and computational resources. BERT-based Speech pre-Training with Random-projection Quantizer (BEST-RQ), is an SSL method that has shown great performance on Automatic Speech Recognition (ASR) while being simpler than other SSL methods, such as wav2vec 2.0. Despite BEST-RQ's great performance, details are lacking in the original paper, such as the amount of GPU/TPU hours used in pre-training, and there is no official easy-to-use open-source implementation. Furthermore, BEST-RQ has not been evaluated on other downstream tasks aside from ASR and speech translation. In this work, we describe a re-implementation of a Random-projection quantizer and perform a preliminary study with a comparison to wav2vec 2.0 on four downstream tasks. We discuss the details and differences of our implementation. We show that a random projection quantizer can achieve similar downstream performance as wav2vec 2.0 while decreasing training time by over a factor of two.
Paper Structure (12 sections, 1 figure, 3 tables)

This paper contains 12 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. wav2vec 2.0 vs. BEST-RQ
  4. Experiments
  5. Pre-training Setting
  6. Model Settings
  7. Downstream Tasks
  8. Results
  9. MP3S Downstream Tasks and Fine-Tuning
  10. Impact of the Masking Ratio
  11. Discussion and Conclusion
  12. Acknowledgment

Figures (1)

  • Figure 1: Diagrams of our BEST-RQ and wav2vec 2.0 architecture. BEST-RQ operates on mel-spectrograms, uses a static quantizer and conformer layers. On the other hand, wav2vec 2.0 operates on raw audio, trains the quantizer, and uses transformer layers.