Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answering

Chun-Yi Kuan, Kai-Wei Chang, Hung-yi Lee

TL;DR

This paper tackles evaluation for text to audio generation by proposing AQAScore, a framework that verifies semantic alignment via audio question answering rather than embedding similarity. It computes a probabilistic score from the Yes/No responses to targeted questions about the audio content, making AQAScore backbone-agnostic and dependent on the reasoning capabilities of audio-aware LLMs. Across human-rated relevance, pairwise preferences, and compositional reasoning benchmarks, AQAScore shows higher correlation with human judgments than CLAPScore and prompting baselines, and its signal improves with stronger backbones. The work discusses limitations and suggests extensions to multi dimensional evaluation and potential reward-based optimization.

Abstract

Although text-to-audio generation has made remarkable progress in realism and diversity, the development of evaluation metrics has not kept pace. Widely-adopted approaches, typically based on embedding similarity like CLAPScore, effectively measure general relevance but remain limited in fine-grained semantic alignment and compositional reasoning. To address this, we introduce AQAScore, a backbone-agnostic evaluation framework that leverages the reasoning capabilities of audio-aware large language models (ALLMs). AQAScore reformulates assessment as a probabilistic semantic verification task; rather than relying on open-ended text generation, it estimates alignment by computing the exact log-probability of a "Yes" answer to targeted semantic queries. We evaluate AQAScore across multiple benchmarks, including human-rated relevance, pairwise comparison, and compositional reasoning tasks. Experimental results show that AQAScore consistently achieves higher correlation with human judgments than similarity-based metrics and generative prompting baselines, showing its effectiveness in capturing subtle semantic inconsistencies and scaling with the capability of underlying ALLMs.

AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answering

TL;DR

This paper tackles evaluation for text to audio generation by proposing AQAScore, a framework that verifies semantic alignment via audio question answering rather than embedding similarity. It computes a probabilistic score from the Yes/No responses to targeted questions about the audio content, making AQAScore backbone-agnostic and dependent on the reasoning capabilities of audio-aware LLMs. Across human-rated relevance, pairwise preferences, and compositional reasoning benchmarks, AQAScore shows higher correlation with human judgments than CLAPScore and prompting baselines, and its signal improves with stronger backbones. The work discusses limitations and suggests extensions to multi dimensional evaluation and potential reward-based optimization.

Abstract

Although text-to-audio generation has made remarkable progress in realism and diversity, the development of evaluation metrics has not kept pace. Widely-adopted approaches, typically based on embedding similarity like CLAPScore, effectively measure general relevance but remain limited in fine-grained semantic alignment and compositional reasoning. To address this, we introduce AQAScore, a backbone-agnostic evaluation framework that leverages the reasoning capabilities of audio-aware large language models (ALLMs). AQAScore reformulates assessment as a probabilistic semantic verification task; rather than relying on open-ended text generation, it estimates alignment by computing the exact log-probability of a "Yes" answer to targeted semantic queries. We evaluate AQAScore across multiple benchmarks, including human-rated relevance, pairwise comparison, and compositional reasoning tasks. Experimental results show that AQAScore consistently achieves higher correlation with human judgments than similarity-based metrics and generative prompting baselines, showing its effectiveness in capturing subtle semantic inconsistencies and scaling with the capability of underlying ALLMs.
Paper Structure (23 sections, 5 equations, 2 figures, 36 tables)

This paper contains 23 sections, 5 equations, 2 figures, 36 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Methods
  4. Experimental Setups
  5. Backbone Models for AQAScore
  6. Benchmark Datasets
  7. Baseline Methods
  8. Evaluation Metrics
  9. Experimental Results
  10. Human-rated Text Relevance
  11. Pairwise Comparison
  12. Compositional Reasoning Analysis
  13. Effect of Question Templates
  14. Discussion
  15. Conclusion
  16. ...and 8 more sections

Figures (2)

  • Figure 1: Overview of AQAScore framework.
  • Figure 2: Robustness of AQAScore with Qwen2.5-Omni-7B across diverse question templates. Error bars indicate the standard deviation over prompt variations. The consistently low variance across benchmarks and metrics suggests that the model’s judgments are stable with respect to prompt phrasing.