Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Analysis of Self-Supervised Speech Models on Children's Speech and Infant Vocalizations

Jialu Li, Mark Hasegawa-Johnson, Nancy L. McElwain

TL;DR

This work probes self-supervised speech models (Wav2Vec 2.0 and HuBERT) to understand how they encode children's speech and infant vocalizations. By evaluating phoneme recognition across adults, older children (8–10), and younger children (1–4), and infant vocalization classification (cry/fuss/babble), the authors reveal that pre-training data and fine-tuning strategies shape layer-level phonetic and paralinguistic representations. Middle SSL layers tend to carry richer phonetic information in adult and older-child speech, while younger children's phonetics are not well captured by some pre-trained models, unless fine-tuning adapts to the domain. For infant VC, pre-training on home recordings helps preserve paralinguistic cues (energy, MFCC, pitch) that boost classification when fine-tuned appropriately, highlighting the impact of pre-training data on downstream pediatric tasks. Overall, the study provides actionable insights into SSL representation dynamics across ages and modalities, informing model design for pediatric speech technologies and developmental science.

Abstract

To understand why self-supervised learning (SSL) models have empirically achieved strong performances on several speech-processing downstream tasks, numerous studies have focused on analyzing the encoded information of the SSL layer representations in adult speech. Limited work has investigated how pre-training and fine-tuning affect SSL models encoding children's speech and vocalizations. In this study, we aim to bridge this gap by probing SSL models on two relevant downstream tasks: (1) phoneme recognition (PR) on the speech of adults, older children (8-10 years old), and younger children (1-4 years old), and (2) vocalization classification (VC) distinguishing cry, fuss, and babble for infants under 14 months old. For younger children's PR, the superiority of fine-tuned SSL models is largely due to their ability to learn features that represent older children's speech and then adapt those features to the speech of younger children. For infant VC, SSL models pre-trained on large-scale home recordings learn to leverage phonetic representations at middle layers, and thereby enhance the performance of this task.

Analysis of Self-Supervised Speech Models on Children's Speech and Infant Vocalizations

TL;DR

This work probes self-supervised speech models (Wav2Vec 2.0 and HuBERT) to understand how they encode children's speech and infant vocalizations. By evaluating phoneme recognition across adults, older children (8–10), and younger children (1–4), and infant vocalization classification (cry/fuss/babble), the authors reveal that pre-training data and fine-tuning strategies shape layer-level phonetic and paralinguistic representations. Middle SSL layers tend to carry richer phonetic information in adult and older-child speech, while younger children's phonetics are not well captured by some pre-trained models, unless fine-tuning adapts to the domain. For infant VC, pre-training on home recordings helps preserve paralinguistic cues (energy, MFCC, pitch) that boost classification when fine-tuned appropriately, highlighting the impact of pre-training data on downstream pediatric tasks. Overall, the study provides actionable insights into SSL representation dynamics across ages and modalities, informing model design for pediatric speech technologies and developmental science.

Abstract

To understand why self-supervised learning (SSL) models have empirically achieved strong performances on several speech-processing downstream tasks, numerous studies have focused on analyzing the encoded information of the SSL layer representations in adult speech. Limited work has investigated how pre-training and fine-tuning affect SSL models encoding children's speech and vocalizations. In this study, we aim to bridge this gap by probing SSL models on two relevant downstream tasks: (1) phoneme recognition (PR) on the speech of adults, older children (8-10 years old), and younger children (1-4 years old), and (2) vocalization classification (VC) distinguishing cry, fuss, and babble for infants under 14 months old. For younger children's PR, the superiority of fine-tuned SSL models is largely due to their ability to learn features that represent older children's speech and then adapt those features to the speech of younger children. For infant VC, SSL models pre-trained on large-scale home recordings learn to leverage phonetic representations at middle layers, and thereby enhance the performance of this task.
Paper Structure (12 sections, 3 figures, 3 tables)

This paper contains 12 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Phoneme Recognition Analysis
  3. Data
  4. Experimental Setting
  5. Canonical Correlation Analysis for Phoneme Labels
  6. Results & Analysis
  7. Infant Vocalization Analysis
  8. Data & Methodology
  9. Infant VC Results & Analysis
  10. CCA for Paralinguistic Features
  11. Conclusion
  12. Acknowledgement

Figures (3)

  • Figure 1: CCA scores (y-axis) over 12 layers (x-axis) computed on LibriSpeech, MyST, and Providence datasets for three pre-trained models, (a)-(c), and four fine-tuned W2V2 models, (d)-(g).
  • Figure 2: Weights learned (y-axis) for 12 layers (x-axis) in weighted average layer for four W2V2 models. For each model, weights sum up to 1 across all layers.
  • Figure 3: CCA scores (y-axis) over 12 layers (x-axis) for paralinguistic feature groups computed on W2V2-LL4300h. PT=Pre-train and FT=Fine-tune.