Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

textless-lib: a Library for Textless Spoken Language Processing

Eugene Kharitonov, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Paden Tomasello, Ann Lee, Ali Elkahky, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, Yossi Adi

TL;DR

Textless-lib addresses the challenge of textless spoken language processing by providing a modular PyTorch library that unifies speech-to-unit encoding, unit-to-unit modeling, and speech synthesis. It offers ready-made encoders, quantizers, vocoders, datasets, and preprocessing to streamline experiments in speaker probing, speech resynthesis, and speech continuation. The paper demonstrates the library through three use-cases and discusses design choices that keep the API lightweight while enabling cross-disciplinary collaboration between speech and NLP. This work lowers barriers for researching textless representations and facilitates exploration of under-resourced languages and non-lexical speech signals with practical tooling and reusable building blocks.

Abstract

Textless spoken language processing research aims to extend the applicability of standard NLP toolset onto spoken language and languages with few or no textual resources. In this paper, we introduce textless-lib, a PyTorch-based library aimed to facilitate research in this research area. We describe the building blocks that the library provides and demonstrate its usability by discuss three different use-case examples: (i) speaker probing, (ii) speech resynthesis and compression, and (iii) speech continuation. We believe that textless-lib substantially simplifies research the textless setting and will be handful not only for speech researchers but also for the NLP community at large. The code, documentation, and pre-trained models are available at https://github.com/facebookresearch/textlesslib/ .

textless-lib: a Library for Textless Spoken Language Processing

TL;DR

Textless-lib addresses the challenge of textless spoken language processing by providing a modular PyTorch library that unifies speech-to-unit encoding, unit-to-unit modeling, and speech synthesis. It offers ready-made encoders, quantizers, vocoders, datasets, and preprocessing to streamline experiments in speaker probing, speech resynthesis, and speech continuation. The paper demonstrates the library through three use-cases and discusses design choices that keep the API lightweight while enabling cross-disciplinary collaboration between speech and NLP. This work lowers barriers for researching textless representations and facilitates exploration of under-resourced languages and non-lexical speech signals with practical tooling and reusable building blocks.

Abstract

Textless spoken language processing research aims to extend the applicability of standard NLP toolset onto spoken language and languages with few or no textual resources. In this paper, we introduce textless-lib, a PyTorch-based library aimed to facilitate research in this research area. We describe the building blocks that the library provides and demonstrate its usability by discuss three different use-case examples: (i) speaker probing, (ii) speech resynthesis and compression, and (iii) speech continuation. We believe that textless-lib substantially simplifies research the textless setting and will be handful not only for speech researchers but also for the NLP community at large. The code, documentation, and pre-trained models are available at https://github.com/facebookresearch/textlesslib/ .
Paper Structure (13 sections, 4 figures, 4 tables)

This paper contains 13 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Speech to Units
  4. Units to Speech
  5. Unit to Units
  6. Library Overview
  7. Interfaces
  8. Pre-trained Models
  9. Examples
  10. Speaker Probing
  11. Speech Resynthesis
  12. Speech Continuation
  13. Discussion and Future Work

Figures (4)

  • Figure 1: A visual description for textless modeling of spoken language. One can perform language modeling for speech continuations Lakhotia2021 or a direct speech resynthesis Polyak2021.
  • Figure 2: We represent speech as three aligned, synchronised streams: discrete pseudo-units, duration, and pitch.
  • Figure 3: textless-lib provides an "encoded" view for standard datasets, such as LibriSpeech.
  • Figure 4: Fully functioning code for discrete audio resynthesis. An audio file is loaded, coverted into a sequence of pseudo-units and transformed back into audio with Tacotron2. The model setup code will download required checkpoints and cache them locally.