Confidence-based Filtering for Speech Dataset Curation with Generative Speech Enhancement Using Discrete Tokens

TL;DR

This paper addresses hallucination in discrete-token GSE during in-the-wild TTS data curation. It introduces a non-intrusive confidence score $S_{utt}$ derived from token-level log-probabilities $s_t$ to filter enhanced speech without clean references. Using Genhancer with a DAC-based tokenizer/decoder, the authors show that $S_{utt}$ correlates strongly with intrusive SE metrics on the EARS-WHAM dataset and can detect content errors missed by standard non-intrusive filters. They demonstrate practical gains by training TTS models on data curated with $S_{utt}$-based filtering, achieving higher UTMOS/DNSMOS and lower WER compared with baselines. Future work will extend to continuous latent-space confidence measures for GSE beyond discrete-token models.

Abstract

Generative speech enhancement (GSE) models show great promise in producing high-quality clean speech from noisy inputs, enabling applications such as curating noisy text-to-speech (TTS) datasets into high-quality ones. However, GSE models are prone to hallucination errors, such as phoneme omissions and speaker inconsistency, which conventional error filtering based on non-intrusive speech quality metrics often fails to detect. To address this issue, we propose a non-intrusive method for filtering hallucination errors from discrete token-based GSE models. Our method leverages the log-probabilities of generated tokens as confidence scores to detect potential errors. Experimental results show that the confidence scores strongly correlate with a suite of intrusive SE metrics, and that our method effectively identifies hallucination errors missed by conventional filtering methods. Furthermore, we demonstrate the practical utility of our method: curating an in-the-wild TTS dataset with our confidence-based filtering improves the performance of subsequently trained TTS models.

Figures (4)

• Figure 1: Overview of the speech dataset curation process with our proposed confidence-based filtering.
• Figure 2: The entire process of our confidence-based filtering. A discrete token-based GSE model outputs enhanced speech $\mathbf{w}_{\text{enhanced}}$, along with token-level confidence scores ($s_1, s_2, \dots, s_T$). If the confidence score $S_{\text{utt}}$ is below a threshold $\tau$, $\mathbf{w}_{\text{enhanced}}$ is filtered out.
• Figure 3: An example of a hallucination error caused by GSE, showing the spectrogram of clean speech (left) and enhanced speech (right).
• Figure 4: Average quality of the filtered dataset vs. acceptance rate for each filtering method with varying filtering thresholds.