Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AudioEval: Automatic Dual-Perspective and Multi-Dimensional Evaluation of Text-to-Audio-Generation

Hui Wang, Jinghua Zhao, Cheng Liu, Yuhang Jia, Haoqin Sun, Jiaming Zhou, Yong Qin

TL;DR

AudioEval tackles the evaluation bottleneck in text-to-audio generation by releasing the first large-scale, dual-perspective, multi-dimensional dataset and a distribution-aware scoring approach. The proposed Qwen-DisQA jointly processes text prompts and generated audio to predict rating distributions across five perceptual dimensions for expert and non-expert viewpoints, using a loss that combines $D_{KL}$ distribution alignment with mean regression. Experimental results show that Qwen-DisQA achieves superior correlations and robustness, particularly at the system level, compared with baselines, enabling scalable, nuanced evaluation of TTA systems. Overall, the work delivers a valuable dataset and a practical evaluation method that can accelerate progress in perceptual quality assessment for TTA research.

Abstract

Text-to-audio (TTA) is rapidly advancing, with broad potential in virtual reality, accessibility, and creative media. However, evaluating TTA quality remains difficult: human ratings are costly and limited, while existing objective metrics capture only partial aspects of perceptual quality. To address this gap, we introduce AudioEval, the first large-scale TTA evaluation dataset, containing 4,200 audio samples from 24 systems with 126,000 ratings across five perceptual dimensions, annotated by both experts and non-experts. Based on this resource, we propose Qwen-DisQA, a multimodal scoring model that jointly processes text prompts and generated audio to predict human-like quality ratings. Experiments show its effectiveness in providing reliable and scalable evaluation. The dataset will be made publicly available to accelerate future research.

AudioEval: Automatic Dual-Perspective and Multi-Dimensional Evaluation of Text-to-Audio-Generation

TL;DR

AudioEval tackles the evaluation bottleneck in text-to-audio generation by releasing the first large-scale, dual-perspective, multi-dimensional dataset and a distribution-aware scoring approach. The proposed Qwen-DisQA jointly processes text prompts and generated audio to predict rating distributions across five perceptual dimensions for expert and non-expert viewpoints, using a loss that combines distribution alignment with mean regression. Experimental results show that Qwen-DisQA achieves superior correlations and robustness, particularly at the system level, compared with baselines, enabling scalable, nuanced evaluation of TTA systems. Overall, the work delivers a valuable dataset and a practical evaluation method that can accelerate progress in perceptual quality assessment for TTA research.

Abstract

Text-to-audio (TTA) is rapidly advancing, with broad potential in virtual reality, accessibility, and creative media. However, evaluating TTA quality remains difficult: human ratings are costly and limited, while existing objective metrics capture only partial aspects of perceptual quality. To address this gap, we introduce AudioEval, the first large-scale TTA evaluation dataset, containing 4,200 audio samples from 24 systems with 126,000 ratings across five perceptual dimensions, annotated by both experts and non-experts. Based on this resource, we propose Qwen-DisQA, a multimodal scoring model that jointly processes text prompts and generated audio to predict human-like quality ratings. Experiments show its effectiveness in providing reliable and scalable evaluation. The dataset will be made publicly available to accelerate future research.
Paper Structure (15 sections, 3 equations, 4 figures, 2 tables)

This paper contains 15 sections, 3 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. AudioEval Dataset
  3. Data Collection
  4. Annotation Protocol
  5. Dataset Statistics and Analysis
  6. Proposed Method
  7. Problem Formulation
  8. Model Overview
  9. Target Distribution
  10. Training Targets
  11. Experiments
  12. Experimental Details
  13. Compared Approaches
  14. Results
  15. Conclusion

Figures (4)

  • Figure 1: Top: score distributions of expert and non-expert raters across five evaluation dimensions. Bottom: correlations between expert and non-expert scores at the clip level (individual utterances) and the system level (per-system averages).
  • Figure 2: Prompt characteristics. Left: distribution of prompt lengths. Right: t-SNE visualization of TF-IDF features.
  • Figure 3: Overview of Qwen-DisQA for TTA quality assessment, trained with distributional alignment.
  • Figure 4: Performance of Qwen-DisQA at different level.