Hallucination in Perceptual Metric-Driven Speech Enhancement Networks
George Close, Thomas Hain, Stefan Goetze
TL;DR
The paper investigates the risk that speech enhancement systems trained to maximize non-intrusive MOS predictors can generate hallucinations that fool the predictor without improving human-perceived quality. It introduces a non-intrusive MOS predictor based on Whisper-derived features and a DPT-FSNet SE system that combines spectral and predictor-based losses, controlled by . Experiments reveal that strong reliance on the predictor loss leads to hallucinations, worsens intrusive metrics, and produces perceptible artefacts in listening tests, while small amounts of spectral loss can mitigate some issues. The findings highlight a misalignment between predictor-driven optimization and human perception, arguing for more robust predictors and evaluation frameworks in real-world deployment.
Abstract
Within the area of speech enhancement, there is an ongoing interest in the creation of neural systems which explicitly aim to improve the perceptual quality of the processed audio. In concert with this is the topic of non-intrusive (i.e. without clean reference) speech quality prediction, for which neural networks are trained to predict human-assigned quality labels directly from distorted audio. When combined, these areas allow for the creation of powerful new speech enhancement systems which can leverage large real-world datasets of distorted audio, by taking inference of a pre-trained speech quality predictor as the sole loss function of the speech enhancement system. This paper aims to identify a potential pitfall with this approach, namely hallucinations which are introduced by the enhancement system `tricking' the speech quality predictor.