Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-Supervised Speech Representations are More Phonetic than Semantic

Kwanghee Choi, Ankita Pasad, Tomohiko Nakamura, Satoru Fukayama, Karen Livescu, Shinji Watanabe

TL;DR

This work interrogates whether self-supervised speech representations encode phonetic or semantic information at the word level. It introduces a carefully curated dataset of synonyms and near-homophones and analyzes layerwise word representations across multiple S3Ms, languages, and pooling/slicing strategies, revealing a consistent phonetic dominance in similarity measures. The study further tests intent classification benchmarks with a bag-of-words baseline, showing that word identity can outperform S3M-based approaches on several datasets, thereby challenging the notion that high IC scores reflect semantic understanding. Collectively, the findings urge a rethinking of semantic evaluations for S3Ms and suggest directions for disentangling phonetic and semantic content in downstream tasks.

Abstract

Self-supervised speech models (S3Ms) have become an effective backbone for speech applications. Various analyses suggest that S3Ms encode linguistic properties. In this work, we seek a more fine-grained analysis of the word-level linguistic properties encoded in S3Ms. Specifically, we curate a novel dataset of near homophone (phonetically similar) and synonym (semantically similar) word pairs and measure the similarities between S3M word representation pairs. Our study reveals that S3M representations consistently and significantly exhibit more phonetic than semantic similarity. Further, we question whether widely used intent classification datasets such as Fluent Speech Commands and Snips Smartlights are adequate for measuring semantic abilities. Our simple baseline, using only the word identity, surpasses S3M-based models. This corroborates our findings and suggests that high scores on these datasets do not necessarily guarantee the presence of semantic content.

Self-Supervised Speech Representations are More Phonetic than Semantic

TL;DR

This work interrogates whether self-supervised speech representations encode phonetic or semantic information at the word level. It introduces a carefully curated dataset of synonyms and near-homophones and analyzes layerwise word representations across multiple S3Ms, languages, and pooling/slicing strategies, revealing a consistent phonetic dominance in similarity measures. The study further tests intent classification benchmarks with a bag-of-words baseline, showing that word identity can outperform S3M-based approaches on several datasets, thereby challenging the notion that high IC scores reflect semantic understanding. Collectively, the findings urge a rethinking of semantic evaluations for S3Ms and suggest directions for disentangling phonetic and semantic content in downstream tasks.

Abstract

Self-supervised speech models (S3Ms) have become an effective backbone for speech applications. Various analyses suggest that S3Ms encode linguistic properties. In this work, we seek a more fine-grained analysis of the word-level linguistic properties encoded in S3Ms. Specifically, we curate a novel dataset of near homophone (phonetically similar) and synonym (semantically similar) word pairs and measure the similarities between S3M word representation pairs. Our study reveals that S3M representations consistently and significantly exhibit more phonetic than semantic similarity. Further, we question whether widely used intent classification datasets such as Fluent Speech Commands and Snips Smartlights are adequate for measuring semantic abilities. Our simple baseline, using only the word identity, surpasses S3M-based models. This corroborates our findings and suggests that high scores on these datasets do not necessarily guarantee the presence of semantic content.
Paper Structure (18 sections, 6 figures, 1 table)

This paper contains 18 sections, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Experimental Setup
  3. Extracting synonyms and near homophones
  4. Datasets
  5. Sampling word pairs
  6. Extracting S3M representations
  7. Findings
  8. Feature slicing squashes representations
  9. Effects of different pooling methods
  10. S3Ms encode more phonetic than semantic content
  11. Crosslingual word pairs follow similar trends
  12. Single speaker utterances also follow similar trends
  13. Is Word Identity Sufficient for Intent Classification?
  14. Settings
  15. Results
  16. ...and 3 more sections

Figures (6)

  • Figure 1: Cosine similarity between paired word representations of HuBERT-large on the LibriSpeech dev-clean and test-clean subsets. We compare audio slicing (top) vs. feature slicing (bottom). Normalized similarity curves (subtracting the Random baseline, right) are included to visualize the differences more clearly.
  • Figure 2: Different pooling methods for HuBERT-large representations.
  • Figure 3: Similarities between word representations of various models.
  • Figure 4: Similarities between crosslingual word representations of WavLM-Large and XLS-R-300M on MSW.
  • Figure 5: HuBERT representations of five speakers' utterances.
  • ...and 1 more figures