Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diff-ETS: Learning a Diffusion Probabilistic Model for Electromyography-to-Speech Conversion

Zhao Ren, Kevin Scheck, Qinhan Hou, Stefano van Gogh, Michael Wand, Tanja Schultz

TL;DR

The proposed Diff-ETS, an ETS model which uses a score-based diffusion probabilistic model to enhance the naturalness of synthesised speech, significantly improved speech naturalness over the baseline.

Abstract

Electromyography-to-Speech (ETS) conversion has demonstrated its potential for silent speech interfaces by generating audible speech from Electromyography (EMG) signals during silent articulations. ETS models usually consist of an EMG encoder which converts EMG signals to acoustic speech features, and a vocoder which then synthesises the speech signals. Due to an inadequate amount of available data and noisy signals, the synthesised speech often exhibits a low level of naturalness. In this work, we propose Diff-ETS, an ETS model which uses a score-based diffusion probabilistic model to enhance the naturalness of synthesised speech. The diffusion model is applied to improve the quality of the acoustic features predicted by an EMG encoder. In our experiments, we evaluated fine-tuning the diffusion model on predictions of a pre-trained EMG encoder, and training both models in an end-to-end fashion. We compared Diff-ETS with a baseline ETS model without diffusion using objective metrics and a listening test. The results indicated the proposed Diff-ETS significantly improved speech naturalness over the baseline.

Diff-ETS: Learning a Diffusion Probabilistic Model for Electromyography-to-Speech Conversion

TL;DR

The proposed Diff-ETS, an ETS model which uses a score-based diffusion probabilistic model to enhance the naturalness of synthesised speech, significantly improved speech naturalness over the baseline.

Abstract

Electromyography-to-Speech (ETS) conversion has demonstrated its potential for silent speech interfaces by generating audible speech from Electromyography (EMG) signals during silent articulations. ETS models usually consist of an EMG encoder which converts EMG signals to acoustic speech features, and a vocoder which then synthesises the speech signals. Due to an inadequate amount of available data and noisy signals, the synthesised speech often exhibits a low level of naturalness. In this work, we propose Diff-ETS, an ETS model which uses a score-based diffusion probabilistic model to enhance the naturalness of synthesised speech. The diffusion model is applied to improve the quality of the acoustic features predicted by an EMG encoder. In our experiments, we evaluated fine-tuning the diffusion model on predictions of a pre-trained EMG encoder, and training both models in an end-to-end fashion. We compared Diff-ETS with a baseline ETS model without diffusion using objective metrics and a listening test. The results indicated the proposed Diff-ETS significantly improved speech naturalness over the baseline.
Paper Structure (14 sections, 5 equations, 2 figures, 1 table)

This paper contains 14 sections, 5 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. EMG Encoder and Speech Vocoder
  4. Score-based Diffusion Probabilistic Model
  5. Forward process
  6. Reverse process
  7. Model Training
  8. EMG encoder
  9. Diffusion probabilistic model
  10. Experiments
  11. Database
  12. Experimental Settings
  13. Results and Discussion
  14. Conclusions and Future Work

Figures (2)

  • Figure 1: The proposed Diff-ETS framework for ETS. The deep blue blocks are trainable and the light blue block of the vocoder is frozen. ResBlock: Residual blocks, Attn: Attention, Conv: Convolutional layers.
  • Figure 2: The LSD and FAD of the two Diff-ETS models were evaluated on the test set. The number of reverse time steps $T$ is $50$, and the temperature $\theta$ varies from $1.0$ to $5.0$. LSD: log-spectral distance, FAD: Frecht audio distance.