Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Role of Prosodic and Lexical Cues in Turn-Taking with Self-Supervised Speech Representations

Sam OConnor Russell, Delphine Charuau, Naomi Harte

TL;DR

The work investigates whether self-supervised speech representations (S3Rs) drive turn-taking using prosodic versus lexical cues and whether these cues are interdependent. By introducing a WORLD vocoder–based method to cleanly manipulate prosody and lexical content, the authors probe the VAP turn-taking model and demonstrate that both cues independently support turn-taking; crucially, prosody alone can nearly match clean-speech performance when lexical content is removed, and vice versa. Results generalize across CPC-based and wav2vec2.0 S3Rs, suggesting the possibility of privacy-preserving, prosody-only turn-taking models and providing a clearer path for SSL interpretability in turn-taking. The study also highlights practical implications for cross-linguistic generalization and future research directions in cue-specific modeling and privacy-aware design.

Abstract

Fluid turn-taking remains a key challenge in human-robot interaction. Self-supervised speech representations (S3Rs) have driven many advances, but it remains unclear whether S3R-based turn-taking models rely on prosodic cues, lexical cues or both. We introduce a vocoder-based approach to control prosody and lexical cues in speech more cleanly than prior work. This allows us to probe the voice-activity projection model, an S3R-based turn-taking model. We find that prediction on prosody-matched, unintelligible noise is similar to accuracy on clean speech. This reveals both prosodic and lexical cues support turn-taking, but either can be used in isolation. Hence, future models may only require prosody, providing privacy and potential performance benefits. When either prosodic or lexical information is disrupted, the model exploits the other without further training, indicating they are encoded in S3Rs with limited interdependence. Results are consistent in CPC-based and wav2vec2.0 S3Rs. We discuss our findings and highlight a number of directions for future work. All code is available to support future research.

The Role of Prosodic and Lexical Cues in Turn-Taking with Self-Supervised Speech Representations

TL;DR

The work investigates whether self-supervised speech representations (S3Rs) drive turn-taking using prosodic versus lexical cues and whether these cues are interdependent. By introducing a WORLD vocoder–based method to cleanly manipulate prosody and lexical content, the authors probe the VAP turn-taking model and demonstrate that both cues independently support turn-taking; crucially, prosody alone can nearly match clean-speech performance when lexical content is removed, and vice versa. Results generalize across CPC-based and wav2vec2.0 S3Rs, suggesting the possibility of privacy-preserving, prosody-only turn-taking models and providing a clearer path for SSL interpretability in turn-taking. The study also highlights practical implications for cross-linguistic generalization and future research directions in cue-specific modeling and privacy-aware design.

Abstract

Fluid turn-taking remains a key challenge in human-robot interaction. Self-supervised speech representations (S3Rs) have driven many advances, but it remains unclear whether S3R-based turn-taking models rely on prosodic cues, lexical cues or both. We introduce a vocoder-based approach to control prosody and lexical cues in speech more cleanly than prior work. This allows us to probe the voice-activity projection model, an S3R-based turn-taking model. We find that prediction on prosody-matched, unintelligible noise is similar to accuracy on clean speech. This reveals both prosodic and lexical cues support turn-taking, but either can be used in isolation. Hence, future models may only require prosody, providing privacy and potential performance benefits. When either prosodic or lexical information is disrupted, the model exploits the other without further training, indicating they are encoded in S3Rs with limited interdependence. Results are consistent in CPC-based and wav2vec2.0 S3Rs. We discuss our findings and highlight a number of directions for future work. All code is available to support future research.
Paper Structure (10 sections, 4 figures, 4 tables)

This paper contains 10 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Self-supervised speech representations and turn-taking
  3. Isolating Prosodic and Lexical Cues
  4. Exploring self-supervised representations in turn-taking
  5. Turn-taking in clean and manipulated speech
  6. Effect of prosodic manipulation and noise
  7. Prosody and lexical cues in turn-taking prediction
  8. Generalisation to wav2vec2.0
  9. Discussion of Key Findings
  10. Conclusion

Figures (4)

  • Figure 1: Spectrograms utterance "would you feel comfortable". Top=original, middle=noise+prosody preserved, bottom=intelligible+prosody flat. Pitch=blue, intensity=yellow.
  • Figure 2: VAP model performance (S/H-Pred metric, 5-fold average $\pm 95\%$ c.i.) trained on clean speech (A) and a mixture of clean and manipulated speech (B). Balanced accuracy for each speech manipulation by test set signal-to-noise ratio (SNR).
  • Figure 3: Word error rate for each manipulation (WER, clamped to 100%) at each SNR in decibels (dB). P=pitch, I=intensity. Mean per speaker $\pm 95\%$ c.i. Legend as above.
  • Figure 4: Percentage of clean speech model accuracy achieved by when tested (top) and trained (bottom) on prosody-matched noise and speech with flat prosody.