Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The ART of Conversation: Measuring Phonetic Convergence and Deliberate Imitation in L2-Speech with a Siamese RNN

Zheng Yuan, Aldo Pastore, Dorina de Jong, Hao Xu, Luciano Fadiga, Alessandro D'Ausilio

TL;DR

This work tackles measuring phonetic convergence in L2-L2 speech by introducing a Siamese RNN that learns speaker- and utterance-level embeddings to quantify holistic spectral similarity. The ART dataset is extended with Slovak L2-English speakers to evaluate scalability and resilience to L1-induced variability, and the model is trained as a binary speaker verification task. Results show the approach captures convergence dynamics, with higher intra-speaker similarity in solo and decreased similarity under interaction and imitation, and a positive link between imitation ability and convergence (r = 0.51, p = 0.0005). Pretraining on a large corpus (VCTK) and a compact Bi-RNN architecture yield robust performance across dialect groups, suggesting a scalable, text-independent method for assessing imitation and convergence in multilingual L2 speech.

Abstract

Phonetic convergence describes the automatic and unconscious speech adaptation of two interlocutors in a conversation. This paper proposes a Siamese recurrent neural network (RNN) architecture to measure the convergence of the holistic spectral characteristics of speech sounds in an L2-L2 interaction. We extend an alternating reading task (the ART) dataset by adding 20 native Slovak L2 English speakers. We train and test the Siamese RNN model to measure phonetic convergence of L2 English speech from three different native language groups: Italian (9 dyads), French (10 dyads) and Slovak (10 dyads). Our results indicate that the Siamese RNN model effectively captures the dynamics of phonetic convergence and the speaker's imitation ability. Moreover, this text-independent model is scalable and capable of handling L1-induced speaker variability.

The ART of Conversation: Measuring Phonetic Convergence and Deliberate Imitation in L2-Speech with a Siamese RNN

TL;DR

This work tackles measuring phonetic convergence in L2-L2 speech by introducing a Siamese RNN that learns speaker- and utterance-level embeddings to quantify holistic spectral similarity. The ART dataset is extended with Slovak L2-English speakers to evaluate scalability and resilience to L1-induced variability, and the model is trained as a binary speaker verification task. Results show the approach captures convergence dynamics, with higher intra-speaker similarity in solo and decreased similarity under interaction and imitation, and a positive link between imitation ability and convergence (r = 0.51, p = 0.0005). Pretraining on a large corpus (VCTK) and a compact Bi-RNN architecture yield robust performance across dialect groups, suggesting a scalable, text-independent method for assessing imitation and convergence in multilingual L2 speech.

Abstract

Phonetic convergence describes the automatic and unconscious speech adaptation of two interlocutors in a conversation. This paper proposes a Siamese recurrent neural network (RNN) architecture to measure the convergence of the holistic spectral characteristics of speech sounds in an L2-L2 interaction. We extend an alternating reading task (the ART) dataset by adding 20 native Slovak L2 English speakers. We train and test the Siamese RNN model to measure phonetic convergence of L2 English speech from three different native language groups: Italian (9 dyads), French (10 dyads) and Slovak (10 dyads). Our results indicate that the Siamese RNN model effectively captures the dynamics of phonetic convergence and the speaker's imitation ability. Moreover, this text-independent model is scalable and capable of handling L1-induced speaker variability.
Paper Structure (13 sections, 4 equations, 4 figures, 2 tables)

This paper contains 13 sections, 4 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Alternating Reading Task dataset
  3. Participants
  4. Task description
  5. Siamese RNN model
  6. Speaker verification task
  7. Data preparation
  8. Training Details
  9. Results
  10. Phonetic convergence measurement
  11. Discussions
  12. Conclusions
  13. Acknowledgements

Figures (4)

  • Figure 1: The Alternating Reading Task. Participants speak when their computer screen shows black sentences. (A) The solo condition; (B) the interactive condition - synonym shown in orange for illustrative purpose only, and (C) the imitation condition.
  • Figure 2: (A) The proposed voice representation module consists of an MFCC extraction module (purple), a bi-directional RNN layer (green), and a feed-forward layer (blue). It generates a low-dimensional vector representation for input voice audio. (B) shows our model pipeline. The model inputs two voice audios and computes the distance between their voice embeddings from two tied-weight voice representation modules.
  • Figure 3: The cosine similarity scores across the solo, interactive and imitation conditions. Subplot (A) shows intra-dyad similarity scores (different speakers) and (B) the intra-speaker similarity scores (same speaker).
  • Figure 4: The correlation between imitation ability and the degree of convergence in interaction with a 95% confidence interval. The blue dots represent speakers. The imitation ability score on the x-axis and the cosine similarity change during interaction on the y-axis were normalized to (0, 1).