Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answering

Chun-Yi Kuan, Hung-yi Lee

TL;DR

AQUA-Bench tackles the underexplored challenge of unanswerability in audio question answering by defining three unanswerable settings—Absent Answer Detection (AAD), Incompatible Answer Set Detection (IASD), and Incompatible Audio Question Detection (IAQD)—and constructing test sets across bioacoustic, instrumental, and vocal sounds, augmented from MMAU. The evaluation adopts a two-stage protocol that first assesses standard answerable accuracy and then measures conditional accuracy on unanswerable cases, revealing a large gap between answerable performance and the ability to abstain. The study shows that while state-of-the-art audio-language models excel on solvable tasks, they struggle to recognize unanswerable questions, though explicit prompting and Chain-of-Thought reasoning can partially recover this latent capability. These findings motivate more robust, trustworthy audio-language systems and provide a principled benchmark for diagnosing and improving abstention behavior in audio QA.

Abstract

Recent advances in audio-aware large language models have shown strong performance on audio question answering. However, existing benchmarks mainly cover answerable questions and overlook the challenge of unanswerable ones, where no reliable answer can be inferred from the audio. Such cases are common in real-world settings, where questions may be misleading, ill-posed, or incompatible with the information. To address this gap, we present AQUA-Bench, a benchmark for Audio Question Unanswerability Assessment. It systematically evaluates three scenarios: Absent Answer Detection (the correct option is missing), Incompatible Answer Set Detection (choices are categorically mismatched with the question), and Incompatible Audio Question Detection (the question is irrelevant or lacks sufficient grounding in the audio). By assessing these cases, AQUA-Bench offers a rigorous measure of model reliability and promotes the development of audio-language systems that are more robust and trustworthy. Our experiments suggest that while models excel on standard answerable tasks, they often face notable challenges with unanswerable ones, pointing to a blind spot in current audio-language understanding.

AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answering

TL;DR

AQUA-Bench tackles the underexplored challenge of unanswerability in audio question answering by defining three unanswerable settings—Absent Answer Detection (AAD), Incompatible Answer Set Detection (IASD), and Incompatible Audio Question Detection (IAQD)—and constructing test sets across bioacoustic, instrumental, and vocal sounds, augmented from MMAU. The evaluation adopts a two-stage protocol that first assesses standard answerable accuracy and then measures conditional accuracy on unanswerable cases, revealing a large gap between answerable performance and the ability to abstain. The study shows that while state-of-the-art audio-language models excel on solvable tasks, they struggle to recognize unanswerable questions, though explicit prompting and Chain-of-Thought reasoning can partially recover this latent capability. These findings motivate more robust, trustworthy audio-language systems and provide a principled benchmark for diagnosing and improving abstention behavior in audio QA.

Abstract

Recent advances in audio-aware large language models have shown strong performance on audio question answering. However, existing benchmarks mainly cover answerable questions and overlook the challenge of unanswerable ones, where no reliable answer can be inferred from the audio. Such cases are common in real-world settings, where questions may be misleading, ill-posed, or incompatible with the information. To address this gap, we present AQUA-Bench, a benchmark for Audio Question Unanswerability Assessment. It systematically evaluates three scenarios: Absent Answer Detection (the correct option is missing), Incompatible Answer Set Detection (choices are categorically mismatched with the question), and Incompatible Audio Question Detection (the question is irrelevant or lacks sufficient grounding in the audio). By assessing these cases, AQUA-Bench offers a rigorous measure of model reliability and promotes the development of audio-language systems that are more robust and trustworthy. Our experiments suggest that while models excel on standard answerable tasks, they often face notable challenges with unanswerable ones, pointing to a blind spot in current audio-language understanding.
Paper Structure (18 sections, 1 figure, 1 table)

This paper contains 18 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Method
  3. Task Formulation
  4. Unanswerable Case Design
  5. Dataset Construction
  6. Evaluation Protocol
  7. Experimental Setups
  8. Result
  9. Performance on Answerable Questions
  10. Degradation on Unanswerable Questions
  11. Analysis of Unanswerable Scenarios
  12. Reasoning Models on Unanswerables
  13. Conclusion
  14. Discussion on Methodology and Limitations
  15. Rationale for Experimental Design
  16. ...and 3 more sections

Figures (1)

  • Figure 1: Overview of AQUA-Bench. It compares standard audio question answering with three unanswerable scenarios: Absent Answer Detection (AAD), Incompatible Answer Set Detection (IASD), and Incompatible Audio Question Detection (IAQD), to evaluate whether ALLMs can recognize and handle unanswerable cases.