Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimizing Byte-level Representation for End-to-end ASR

Roger Hsiao, Liuhui Deng, Erik McDermott, Ruchir Travadi, Xiaodan Zhuang

TL;DR

In an English/Mandarin dictation task, it is shown that a bilingual ASR model built with this approach can outperform UTF-8 representation by 5% relative in error rate.

Abstract

We propose a novel approach to optimizing a byte-level representation for end-to-end automatic speech recognition (ASR). Byte-level representation is often used by large scale multilingual ASR systems when the character set of the supported languages is large. The compactness and universality of byte-level representation allow the ASR models to use smaller output vocabularies and therefore, provide more flexibility. UTF-8 is a commonly used byte-level representation for multilingual ASR, but it is not designed to optimize machine learning tasks directly. By using auto-encoder and vector quantization, we show that we can optimize a byte-level representation for ASR and achieve better accuracy. Our proposed framework can incorporate information from different modalities, and provides an error correction mechanism. In an English/Mandarin dictation task, we show that a bilingual ASR model built with this approach can outperform UTF-8 representation by 5% relative in error rate.

Optimizing Byte-level Representation for End-to-end ASR

TL;DR

In an English/Mandarin dictation task, it is shown that a bilingual ASR model built with this approach can outperform UTF-8 representation by 5% relative in error rate.

Abstract

We propose a novel approach to optimizing a byte-level representation for end-to-end automatic speech recognition (ASR). Byte-level representation is often used by large scale multilingual ASR systems when the character set of the supported languages is large. The compactness and universality of byte-level representation allow the ASR models to use smaller output vocabularies and therefore, provide more flexibility. UTF-8 is a commonly used byte-level representation for multilingual ASR, but it is not designed to optimize machine learning tasks directly. By using auto-encoder and vector quantization, we show that we can optimize a byte-level representation for ASR and achieve better accuracy. Our proposed framework can incorporate information from different modalities, and provides an error correction mechanism. In an English/Mandarin dictation task, we show that a bilingual ASR model built with this approach can outperform UTF-8 representation by 5% relative in error rate.
Paper Structure (12 sections, 5 equations, 1 figure, 3 tables, 1 algorithm)

This paper contains 12 sections, 5 equations, 1 figure, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. UTF-8 based representation
  3. Optimizing byte-level representation
  4. Formulation as a latent variable optimization problem
  5. Formulation as auto-encoding optimization problem
  6. Vector quantization variational auto-encoder
  7. Convert bytes to labels with error correction
  8. Comparing UTF-8 and VQ based representation
  9. Experimental results
  10. Ablation study
  11. Conclusions and future work
  12. Acknowledgment

Figures (1)

  • Figure 1: Overview of the proposed auto-encoder. The label tokens could be words/subwords/characters. In our experiments, the label tokens are characters.