Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bigger is not Always Better: The Effect of Context Size on Speech Pre-Training

Sean Robertson, Ewan Dunbar

TL;DR

This work investigates how much context is necessary to achieve high-quality pre-trained acoustic models using self-supervised learning and adapts contrastive predictive coding (CPC), which is adapted to be able to precisely control the amount of context visible to the model during training and inference.

Abstract

It has been generally assumed in the automatic speech recognition (ASR) literature that it is better for models to have access to wider context windows. Yet, many of the potential reasons this might be true in the supervised setting do not necessarily transfer over to the case of unsupervised learning. We investigate how much context is necessary to achieve high-quality pre-trained acoustic models using self-supervised learning. We principally investigate contrastive predictive coding (CPC), which we adapt to be able to precisely control the amount of context visible to the model during training and inference. We find that phone discriminability in the resulting model representations peaks at around 40~ms of preceding context, and that having too much context (beyond around 320 ms) substantially degrades the quality of the representations. Surprisingly, we find that this pattern also transfers to supervised ASR when the pre-trained representations are used as frozen input features. Our results point to potential changes in the design of current upstream architectures to better facilitate a variety of downstream tasks.

Bigger is not Always Better: The Effect of Context Size on Speech Pre-Training

TL;DR

This work investigates how much context is necessary to achieve high-quality pre-trained acoustic models using self-supervised learning and adapts contrastive predictive coding (CPC), which is adapted to be able to precisely control the amount of context visible to the model during training and inference.

Abstract

It has been generally assumed in the automatic speech recognition (ASR) literature that it is better for models to have access to wider context windows. Yet, many of the potential reasons this might be true in the supervised setting do not necessarily transfer over to the case of unsupervised learning. We investigate how much context is necessary to achieve high-quality pre-trained acoustic models using self-supervised learning. We principally investigate contrastive predictive coding (CPC), which we adapt to be able to precisely control the amount of context visible to the model during training and inference. We find that phone discriminability in the resulting model representations peaks at around 40~ms of preceding context, and that having too much context (beyond around 320 ms) substantially degrades the quality of the representations. Surprisingly, we find that this pattern also transfers to supervised ASR when the pre-trained representations are used as frozen input features. Our results point to potential changes in the design of current upstream architectures to better facilitate a variety of downstream tasks.
Paper Structure (14 sections, 7 equations, 1 figure, 4 tables)

This paper contains 14 sections, 7 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Causal, Chunked Self-Attention
  5. ABX-LS Phone Discrimination Benchmark
  6. CPC and Pre-Training
  7. Experiments
  8. Main Setup
  9. Auxiliary Setups
  10. ASR Task
  11. Results
  12. Discussion
  13. Conclusions
  14. Acknowledgements

Figures (1)

  • Figure 1: Comparing phoneme-level ABX-LS error rates across the various context widths $W$ and varieties of measure. Error bars are $\pm 1$ SE. $N=5$ models were trained per width.