Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spoken Word2Vec: Learning Skipgram Embeddings from Speech

Mohammad Amaan Sayeed, Hanan Aldarmaki

TL;DR

This work addresses the challenge of deriving distributional semantics from spoken words, revealing that shallow two-stage architectures struggle when input units are phonetically correlated. It proposes a deep end-to-end skipgram variant that uses discretized acoustic units derived from self-supervised features to learn semantic relationships at the corpus level. Empirical results show that two-stage approaches primarily capture phonetic or orthographic cues, while end-to-end models—especially with discrete HuBERT-based units and sufficient depth—achieve meaningful semantic alignment, improving with training and scale. The findings highlight the critical roles of input representation, discretization, and model depth in enabling semantic word embeddings from speech, while noting challenges in fully unsupervised segmentation.

Abstract

Text word embeddings that encode distributional semantics work by modeling contextual similarities of frequently occurring words. Acoustic word embeddings, on the other hand, typically encode low-level phonetic similarities. Semantic embeddings for spoken words have been previously explored using analogous algorithms to Word2Vec, but the resulting vectors still mainly encoded phonetic rather than semantic features. In this paper, we examine the assumptions and architectures used in previous works and show experimentally how shallow skipgram-like algorithms fail to encode distributional semantics when the input units are acoustically correlated. We illustrate the potential of an alternative deep end-to-end variant of the model and examine the effects on the resulting embeddings, showing positive results of semantic relatedness in the embedding space.

Spoken Word2Vec: Learning Skipgram Embeddings from Speech

TL;DR

This work addresses the challenge of deriving distributional semantics from spoken words, revealing that shallow two-stage architectures struggle when input units are phonetically correlated. It proposes a deep end-to-end skipgram variant that uses discretized acoustic units derived from self-supervised features to learn semantic relationships at the corpus level. Empirical results show that two-stage approaches primarily capture phonetic or orthographic cues, while end-to-end models—especially with discrete HuBERT-based units and sufficient depth—achieve meaningful semantic alignment, improving with training and scale. The findings highlight the critical roles of input representation, discretization, and model depth in enabling semantic word embeddings from speech, while noting challenges in fully unsupervised segmentation.

Abstract

Text word embeddings that encode distributional semantics work by modeling contextual similarities of frequently occurring words. Acoustic word embeddings, on the other hand, typically encode low-level phonetic similarities. Semantic embeddings for spoken words have been previously explored using analogous algorithms to Word2Vec, but the resulting vectors still mainly encoded phonetic rather than semantic features. In this paper, we examine the assumptions and architectures used in previous works and show experimentally how shallow skipgram-like algorithms fail to encode distributional semantics when the input units are acoustically correlated. We illustrate the potential of an alternative deep end-to-end variant of the model and examine the effects on the resulting embeddings, showing positive results of semantic relatedness in the embedding space.
Paper Structure (13 sections, 3 figures, 1 table)

This paper contains 13 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Word embeddings and subword units
  4. Speech-based word embeddings
  5. Architectures & Scale: Sanity Check
  6. Models
  7. Data, training, and evaluation scheme
  8. Results
  9. Deep skipgram model for spoken words
  10. Acoustic Features
  11. Discrete Units
  12. Results
  13. Discussion & Conclusions

Figures (3)

  • Figure 1: Pearson correlation with target cosine distance and edit distance for different scales (s) of the (a) two-stage model vs. (b) end-to-end model. (c) shows the progress of correlation scores throughout the first 50 epochs of training the end-to-en model with s=4.
  • Figure 2: Character-based skip-gram model consisting of GRU character encoder, followed by a multi-layer linear network to handle continuous input. The encoder can be pre-trained using a reconstruction loss with the GRU decoder as in (a), or end-to-end within (b).
  • Figure 3: Pearson correlation with target cosine distances and edit distances across the 100 training epochs of the end-to-end model with s=4. The input features are obtained by clustering acoustic frames with K=100 using (a) MFCC, (b) Wav2Vec2, and (c) HuBERT features.