Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Simulating Native Speaker Shadowing for Nonnative Speech Assessment with Latent Speech Representations

Haopeng Geng, Daisuke Saito, Nobuaki Minematsu

TL;DR

The paper tackles the challenge of giving accurate, speech-based feedback for nonnative learners by replacing costly native shadowing with a virtual L1-shadowing-L2 system. It combines Seq2Seq voice conversion with latent, self-supervised speech representations to generate L1-shadowing-like utterances from L2 inputs, targeting both $L1_{S1}$ and $L1_{SS}$ as references. Key contributions include two fine-tuning strategies (Encoder and Decoder) and evidence that SSL-based embeddings (notably HuBERT) improve linguistic fidelity and naturalness, with the PM-SS configuration achieving the best results. The work demonstrates promise for shadowing-informed CAPT tools and highlights avenues for future work, such as incorporating explicit priors on $L1_{SS}$-$L1_{S1}$ distance and leveraging parallel L2 data for broader applicability.

Abstract

Evaluating speech intelligibility is a critical task in computer-aided language learning systems. Traditional methods often rely on word error rates (WER) provided by automatic speech recognition (ASR) as intelligibility scores. However, this approach has significant limitations due to notable differences between human speech recognition (HSR) and ASR. A promising alternative is to involve a native (L1) speaker in shadowing what nonnative (L2) speakers say. Breakdowns or mispronunciations in the L1 speaker's shadowing utterance can serve as indicators for assessing L2 speech intelligibility. In this study, we propose a speech generation system that simulates the L1 shadowing process using voice conversion (VC) techniques and latent speech representations. Our experimental results demonstrate that this method effectively replicates the L1 shadowing process, offering an innovative tool to evaluate L2 speech intelligibility. Notably, systems that utilize self-supervised speech representations (S3R) show a higher degree of similarity to real L1 shadowing utterances in both linguistic accuracy and naturalness.

Simulating Native Speaker Shadowing for Nonnative Speech Assessment with Latent Speech Representations

TL;DR

The paper tackles the challenge of giving accurate, speech-based feedback for nonnative learners by replacing costly native shadowing with a virtual L1-shadowing-L2 system. It combines Seq2Seq voice conversion with latent, self-supervised speech representations to generate L1-shadowing-like utterances from L2 inputs, targeting both and as references. Key contributions include two fine-tuning strategies (Encoder and Decoder) and evidence that SSL-based embeddings (notably HuBERT) improve linguistic fidelity and naturalness, with the PM-SS configuration achieving the best results. The work demonstrates promise for shadowing-informed CAPT tools and highlights avenues for future work, such as incorporating explicit priors on - distance and leveraging parallel L2 data for broader applicability.

Abstract

Evaluating speech intelligibility is a critical task in computer-aided language learning systems. Traditional methods often rely on word error rates (WER) provided by automatic speech recognition (ASR) as intelligibility scores. However, this approach has significant limitations due to notable differences between human speech recognition (HSR) and ASR. A promising alternative is to involve a native (L1) speaker in shadowing what nonnative (L2) speakers say. Breakdowns or mispronunciations in the L1 speaker's shadowing utterance can serve as indicators for assessing L2 speech intelligibility. In this study, we propose a speech generation system that simulates the L1 shadowing process using voice conversion (VC) techniques and latent speech representations. Our experimental results demonstrate that this method effectively replicates the L1 shadowing process, offering an innovative tool to evaluate L2 speech intelligibility. Notably, systems that utilize self-supervised speech representations (S3R) show a higher degree of similarity to real L1 shadowing utterances in both linguistic accuracy and naturalness.
Paper Structure (21 sections, 2 equations, 3 figures, 2 tables)

This paper contains 21 sections, 2 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Research background
  3. Native Speech Shadowing
  4. Seq2Seq Voice Conversion
  5. Self Supervised Speech Representations
  6. Purposed Method
  7. Latent Speech Representation Embedding for L2 Speech
  8. Two-step Training Strategies for $L1_{S1}$-targeting System
  9. Encoder Fine Tuning
  10. Decoder Fine Tuning
  11. Experimental Settings
  12. Dataset
  13. Seq2Seq Model
  14. Pretrained Embedding and Decoding Models
  15. Experimental Evaluation
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Native speaker shadowing for non-native speech assessment. $L1_{S1}$ represents a native speaker’s initial shadowing, while $L1_{SS}$ denotes his/her script-shadowing, which is the most fluent shadowing.
  • Figure 2: Overview of the proposed L1-shadowing-L2 system: after the source/target embeddings are processed by their respective encoders, the Seq2Seq model directly maps the source input $L2_{R}$ to the target outputs $L1_{S1}$ or $L1_{SS}$.
  • Figure 3: Overview of two proposed fine-tuning approaches for mapping $L2_{R}$ to $L1_{S1}$ utilizing $L1_{SS}$.