Speaking Without Sound: Multi-speaker Silent Speech Voicing with Facial Inputs Only

TL;DR

The paper tackles silent, multi-speaker speech generation by leveraging silent EMG signals for linguistic content and facial images to imprint target speaker timbre. The authors introduce a pitch-disentangled content embedding and a pitch-flattening module to separate content from speaker pitch, enabling accurate cross-modal voice synthesis without audible inputs. The approach combines a face-conditioned voice conversion network, EMG-based content estimation via DTW-aligned embeddings, and a global pitch estimator, achieving improved intelligibility, speaker consistency, and pitch alignment, with a new Local F0 deviation metric to assess content-related pitch quality. Results show that the Flatten model enhances silent EMG intelligibility and pitch consistency, suggesting practical potential for assistive technologies in speech-impaired users. This work broadens silent speech interface capabilities to multi-speaker scenarios by decoupling content from pitch and relying on facial cues for identity transfer, promoting accessible, identity-consistent voicing without audio data.

Abstract

In this paper, we introduce a novel framework for generating multi-speaker speech without relying on any audible inputs. Our approach leverages silent electromyography (EMG) signals to capture linguistic content, while facial images are used to match with the vocal identity of the target speaker. Notably, we present a pitch-disentangled content embedding that enhances the extraction of linguistic content from EMG signals. Extensive analysis demonstrates that our method can generate multi-speaker speech without any audible inputs and confirms the effectiveness of the proposed pitch-disentanglement approach.

Figures (1)

• Figure 1: Overview of the proposed multi-speaker EMG-to-Speech generation framework. In the (a) Inference phase, content embedding is estimated from EMG signals, while speaker embedding and global pitch information are derived from a facial image. During the (b) Training phase, for training the (i) face-based voice conversion network, a predefined speaker-wise global pitch ($f_{0,\mathit{gt}}^{\mathit{g}}$) is used to estimate frame-wise $F0$ values. However, since the $f_{0,\mathit{gt}}^{\mathit{g}}$ values for unseen target speakers are not available during the inference, a (iii) face-based global pitch estimation network is independently trained using only face images. Additionally, as the speech-based content encoder is not available during inference, the (ii) EMG-based contents estimation network is trained. Each network is trained independently during the training phase.