Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HiFi-Glot: High-Fidelity Neural Formant Synthesis with Differentiable Resonant Filters

Yicheng Gu, Pablo Pérez Zarazaga, Chaoren Wang, Zhizheng Wu, Zofia Malisz, Gustav Eje Henter, Lauri Juvela

TL;DR

Experimental results demonstrate that the proposed HiFi-Glot model can generate speech with higher perceptual quality and naturalness while exhibiting a more precise control over formant frequencies, outperforming industry-standard formant manipulation tools such as Praat.

Abstract

Formant synthesis aims to generate speech with controllable formant structures, enabling precise control of vocal resonance and phonetic features. However, while existing formant synthesis approaches enable precise formant manipulation, they often yield an impoverished speech signal by failing to capture the complex co-occurring acoustic cues essential for naturalness. To address this issue, this letter presents HiFi-Glot, an end-to-end neural formant synthesis system that achieves both precise formant control and high-fidelity speech synthesis. Specifically, the proposed model adopts a source--filter architecture inspired by classical formant synthesis, where a neural vocoder generates the glottal excitation signal, and differentiable resonant filters model the formants to produce the speech waveform. Experiment results demonstrate that our proposed HiFi-Glot model can generate speech with higher perceptual quality and naturalness while exhibiting a more precise control over formant frequencies, outperforming industry-standard formant manipulation tools such as Praat. Code, checkpoints, and representative audio samples are available at https://www.yichenggu.com/HiFi-Glot/.

HiFi-Glot: High-Fidelity Neural Formant Synthesis with Differentiable Resonant Filters

TL;DR

Experimental results demonstrate that the proposed HiFi-Glot model can generate speech with higher perceptual quality and naturalness while exhibiting a more precise control over formant frequencies, outperforming industry-standard formant manipulation tools such as Praat.

Abstract

Formant synthesis aims to generate speech with controllable formant structures, enabling precise control of vocal resonance and phonetic features. However, while existing formant synthesis approaches enable precise formant manipulation, they often yield an impoverished speech signal by failing to capture the complex co-occurring acoustic cues essential for naturalness. To address this issue, this letter presents HiFi-Glot, an end-to-end neural formant synthesis system that achieves both precise formant control and high-fidelity speech synthesis. Specifically, the proposed model adopts a source--filter architecture inspired by classical formant synthesis, where a neural vocoder generates the glottal excitation signal, and differentiable resonant filters model the formants to produce the speech waveform. Experiment results demonstrate that our proposed HiFi-Glot model can generate speech with higher perceptual quality and naturalness while exhibiting a more precise control over formant frequencies, outperforming industry-standard formant manipulation tools such as Praat. Code, checkpoints, and representative audio samples are available at https://www.yichenggu.com/HiFi-Glot/.
Paper Structure (16 sections, 5 equations, 3 figures)

This paper contains 16 sections, 5 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Methodology
  3. Model Architecture
  4. All-Pole Synthesis Filter
  5. Training Targets
  6. Experiments
  7. Experiment Setup
  8. Datasets
  9. Baselines
  10. Configuration
  11. Preprocessing
  12. Training
  13. Evaluation Metrics
  14. Effectiveness on Speech Parameter Manipulation
  15. Effectiveness on Vocal Tract Length Modification
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Architecture and training schemes for HiFi-Glot. The model consists of a GCNN feature mapper, an NSF-HiFiGAN decoder, and four different discriminators. Dark-blue blocks represent trainable neural networks, light-blue blocks represent intermediate features, yellow blocks are differentiable DSP modules, and dashed lines indicate loss-function input pairings.
  • Figure 2: Box plots of speech-parameter manipulation results. Each parameter is scaled over the range $[0.7, \dots, 1.3]$ and then re-synthesized using different systems. Box edges represent 1$^{\text{st}}$ and 3$^{\text{rd}}$ quartiles and whiskers extend to $1.5 \times \text{IQL}$. Note that all metrics are computed on voiced speech segments only, since we only manipulate speech parameter trajectories there. Additionally, Praat has no results for Tilt, Centroid, and Energy as it does not support manipulating these speech parameters.
  • Figure 3: Bar plots of the MUSHRA-like listening test results with 95% confidence intervals (CIs). All formant parameters are scaled over the range $[0.7, \dots, 1.3]$ and then re-synthesized using different systems.