Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecs

Wei-Cheng Tseng, David Harwath

TL;DR

Codec2Vec tackles learning general-purpose speech representations using only discrete audio codec units, achieving substantial storage and compute savings. It pre-extracts codec units, applies a masked prediction objective to discrete inputs, and evaluates on the SUPERB benchmark with a lightweight downstream module. Through reconstruction, iterative clustering, and online clustering targets, clustering-based strategies yield competitive performance relative to continuous-input baselines while delivering up to 16.5x storage reductions and a 2.3x speedup. The work demonstrates the feasibility and practical benefits of fully discrete-input SSL for broad speech tasks and highlights codec design and target strategies as key factors for future improvements.

Abstract

Recent advancements in neural audio codecs have not only enabled superior audio compression but also enhanced speech synthesis techniques. Researchers are now exploring their potential as universal acoustic feature extractors for a broader range of speech processing tasks. Building on this trend, we introduce Codec2Vec, the first speech representation learning framework that relies exclusively on discrete audio codec units. This approach offers several advantages, including improved data storage and transmission efficiency, faster training, and enhanced data privacy. We explore masked prediction with various training target derivation strategies to thoroughly understand the effectiveness of this framework. Evaluated on the SUPERB benchmark, Codec2Vec achieves competitive performance compared to continuous-input models while reducing storage requirements by up to 16.5x and training time by 2.3x, showcasing its scalability and efficiency.

Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecs

TL;DR

Codec2Vec tackles learning general-purpose speech representations using only discrete audio codec units, achieving substantial storage and compute savings. It pre-extracts codec units, applies a masked prediction objective to discrete inputs, and evaluates on the SUPERB benchmark with a lightweight downstream module. Through reconstruction, iterative clustering, and online clustering targets, clustering-based strategies yield competitive performance relative to continuous-input baselines while delivering up to 16.5x storage reductions and a 2.3x speedup. The work demonstrates the feasibility and practical benefits of fully discrete-input SSL for broad speech tasks and highlights codec design and target strategies as key factors for future improvements.

Abstract

Recent advancements in neural audio codecs have not only enabled superior audio compression but also enhanced speech synthesis techniques. Researchers are now exploring their potential as universal acoustic feature extractors for a broader range of speech processing tasks. Building on this trend, we introduce Codec2Vec, the first speech representation learning framework that relies exclusively on discrete audio codec units. This approach offers several advantages, including improved data storage and transmission efficiency, faster training, and enhanced data privacy. We explore masked prediction with various training target derivation strategies to thoroughly understand the effectiveness of this framework. Evaluated on the SUPERB benchmark, Codec2Vec achieves competitive performance compared to continuous-input models while reducing storage requirements by up to 16.5x and training time by 2.3x, showcasing its scalability and efficiency.

Paper Structure

This paper contains 12 sections, 3 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodologies
  4. Using Discrete Inputs from Neural Audio Codecs
  5. Representation Learning with Masked Prediction
  6. Experiments
  7. Downstream Evaluation
  8. Storage and Training Efficiency
  9. Impact of Neural Codec Model
  10. Discussion and Limitations
  11. Conclusions
  12. Acknowledgements

Figures (1)

  • Figure 1: Overall pipeline of Codec2Vec. The pipeline comprises three stages: (a)Pre-computation of discrete codec units for the pre-training dataset using a neural audio codec; (b) Codec2Vec SSL pretraining using masked prediction; (c) after pretraining, fine-tuning with a lightweight downstream module using discretized dataset inputs.