Towards Evaluating Generative Audio: Insights from Neural Audio Codec Embedding Distances
Arijit Biswas, Lars Villemoes
TL;DR
The paper addresses the challenge of scalable, reliable audio quality evaluation for generative audio. It systematically compares Fréchet Audio Distance (FAD) and Maximum Mean Discrepancy (MMD) across neural audio codecs (EnCodec, DAC, and the enhanced DACe) and popular embeddings, using MUSHRA judgments as ground truth. Key findings show that FAD generally yields stronger correlations with human ratings than MMD, and that higher-fidelity NAC embeddings (notably DACe) improve perceptual alignment, while popular embeddings like CLAP-M and OpenL3 can surpass NACs in some settings. The work demonstrates that NAC embeddings offer a practical zero-shot approach to quality assessment that also benefits future codec development, bridging compression and perceptual evaluation.
Abstract
Neural audio codecs (NACs) achieve low-bitrate compression by learning compact audio representations, which can also serve as features for perceptual quality evaluation. We introduce DACe, an enhanced, higher-fidelity version of the Descript Audio Codec (DAC), trained on diverse real and synthetic tonal data with balanced sampling. We systematically compare Fréchet Audio Distance (FAD) and Maximum Mean Discrepancy (MMD) on MUSHRA tests across speech, music, and mixed content. FAD consistently outperforms MMD, and embeddings from higher-fidelity NACs (such as DACe) show stronger correlations with human judgments. While CLAP LAION Music (CLAP-M) and OpenL3 Mel128 (OpenL3-128M) embeddings achieve higher correlations, NAC embeddings provide a practical zero-shot approach to audio quality assessment, requiring only unencoded audio for training. These results demonstrate the dual utility of NACs for compression and perceptually informed audio evaluation.