Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Objective Speech Quality Metrics for Neural Audio Codecs

Luca A. Lanzendörfer, Florian Grötschla

TL;DR

Problem: Assess how well objective metrics predict human-perceived quality for neural audio codecs at low bitrates in speech and mixed audio. Approach: A MUSHRA study with six NACs and two vocoders, paired with a broad set of objective metrics, to measure correlations with subjective scores. Findings: PESQ and SCOREQ show strongest correlation with human judgments across both speech-only and combined audio; several perceptual metrics underperform on NAC distortions and newer metrics excel only in speech-only contexts; simple measures like SNR/SDR can be competitive. Impact: Provides practical guidance for metric selection in NAC evaluation and supports more scalable quality assessment during development.

Abstract

Neural audio codecs have gained recent popularity for their use in generative modeling as they offer high-fidelity audio reconstruction at low bitrates. While human listening studies remain the gold standard for assessing perceptual quality, they are time-consuming and impractical. In this work, we examine the reliability of existing objective quality metrics in assessing the performance of recent neural audio codecs. To this end, we conduct a MUSHRA listening test on high-fidelity speech signals and analyze the correlation between subjective scores and widely used objective metrics. Our results show that, while some metrics align well with human perception, others struggle to capture relevant distortions. Our findings provide practical guidance for selecting appropriate evaluation metrics when using neural audio codecs for speech.

Evaluating Objective Speech Quality Metrics for Neural Audio Codecs

TL;DR

Problem: Assess how well objective metrics predict human-perceived quality for neural audio codecs at low bitrates in speech and mixed audio. Approach: A MUSHRA study with six NACs and two vocoders, paired with a broad set of objective metrics, to measure correlations with subjective scores. Findings: PESQ and SCOREQ show strongest correlation with human judgments across both speech-only and combined audio; several perceptual metrics underperform on NAC distortions and newer metrics excel only in speech-only contexts; simple measures like SNR/SDR can be competitive. Impact: Provides practical guidance for metric selection in NAC evaluation and supports more scalable quality assessment during development.

Abstract

Neural audio codecs have gained recent popularity for their use in generative modeling as they offer high-fidelity audio reconstruction at low bitrates. While human listening studies remain the gold standard for assessing perceptual quality, they are time-consuming and impractical. In this work, we examine the reliability of existing objective quality metrics in assessing the performance of recent neural audio codecs. To this end, we conduct a MUSHRA listening test on high-fidelity speech signals and analyze the correlation between subjective scores and widely used objective metrics. Our results show that, while some metrics align well with human perception, others struggle to capture relevant distortions. Our findings provide practical guidance for selecting appropriate evaluation metrics when using neural audio codecs for speech.

Paper Structure

This paper contains 10 sections, 2 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Models
  4. Objective Metrics
  5. Subjective Listening Test
  6. Correlation Analysis
  7. Results
  8. Subjective Results
  9. Correlation Analysis
  10. Conclusion

Figures (1)

  • Figure 1: MUSHRA listening test results for speech-only (left) and combined audio (right). The shaded area and whiskers represent 95% CI. The reference, 7 kHz anchor, and 3.5 kHz anchor were rated at $98.7\pm0.5$, $40.3\pm3.2$, $18.7\pm2.5$ for speech, and $98.1\pm1.0$, $26.7\pm3.2$, $13.6\pm2.2$ for combined audio, respectively.