Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Representation of perceived prosodic similarity of conversational feedback

Livia Qian, Carol Figueroa, Gabriel Skantze

TL;DR

This study tackles how prosody shapes the perceived meaning of vocal feedback when lexical content is held constant. It systematically compares pitch, spectral, and self-supervised speech embeddings against human judgments of prosodic similarity gathered via triadic comparisons, revealing that SSL embeddings better capture prosody than simple pitch features. The authors further show that contrastive learning can compress these representations into low-dimensional latent spaces that remain aligned with human perception, with middle network layers being most informative for prosody. The findings offer a path to compact, speaker-independent prosodic representations suitable for grounding and downstream tasks in dialogue systems, including emotion and intent recognition, while highlighting challenges posed by multi-speaker data and recording quality.

Abstract

Vocal feedback (e.g., `mhm', `yeah', `okay') is an important component of spoken dialogue and is crucial to ensuring common ground in conversational systems. The exact meaning of such feedback is conveyed through both lexical and prosodic form. In this work, we investigate the perceived prosodic similarity of vocal feedback with the same lexical form, and to what extent existing speech representations reflect such similarities. A triadic comparison task with recruited participants is used to measure perceived similarity of feedback responses taken from two different datasets. We find that spectral and self-supervised speech representations encode prosody better than extracted pitch features, especially in the case of feedback from the same speaker. We also find that it is possible to further condense and align the representations to human perception through contrastive learning.

Representation of perceived prosodic similarity of conversational feedback

TL;DR

This study tackles how prosody shapes the perceived meaning of vocal feedback when lexical content is held constant. It systematically compares pitch, spectral, and self-supervised speech embeddings against human judgments of prosodic similarity gathered via triadic comparisons, revealing that SSL embeddings better capture prosody than simple pitch features. The authors further show that contrastive learning can compress these representations into low-dimensional latent spaces that remain aligned with human perception, with middle network layers being most informative for prosody. The findings offer a path to compact, speaker-independent prosodic representations suitable for grounding and downstream tasks in dialogue systems, including emotion and intent recognition, while highlighting challenges posed by multi-speaker data and recording quality.

Abstract

Vocal feedback (e.g., `mhm', `yeah', `okay') is an important component of spoken dialogue and is crucial to ensuring common ground in conversational systems. The exact meaning of such feedback is conveyed through both lexical and prosodic form. In this work, we investigate the perceived prosodic similarity of vocal feedback with the same lexical form, and to what extent existing speech representations reflect such similarities. A triadic comparison task with recruited participants is used to measure perceived similarity of feedback responses taken from two different datasets. We find that spectral and self-supervised speech representations encode prosody better than extracted pitch features, especially in the case of feedback from the same speaker. We also find that it is possible to further condense and align the representations to human perception through contrastive learning.
Paper Structure (13 sections, 2 equations, 2 figures, 1 table)

This paper contains 13 sections, 2 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. Data
  4. Perception study
  5. Stimuli
  6. Task
  7. Participants
  8. Evaluation of existing representations
  9. Metrics and evaluation of the study
  10. Effect of embedding layers
  11. Aligning representations with contrastive learning
  12. Discussion and Conclusion
  13. Acknowledgements

Figures (2)

  • Figure 1: Effect of layers on agreement with people's choices (%). The layers (0-24) are shown from left to right.
  • Figure 2: Test accuracy of the models trained on speech embeddings with different latent sizes, as expressed with cosine similarity (%). LP + length is the concatenation of the first three LP coefficients and the voiced length, while LP is that of the three coefficients.