Pardon? Evaluating Conversational Repair in Large Audio-Language Models

TL;DR

The paper addresses the gap in evaluating spoken QA models by distinguishing semantic answerability from input, introducing a repair-aware framework that pairs answerable and unanswerable inputs via semantic-acoustic masking. It defines the EAR score as the harmonic mean of task competence under answerable conditions and repair behavior under unanswerable conditions, $\mathrm{EAR} = \frac{2 \cdot C \cdot R}{C + R}$, and validates it on WDYL and SLUE-SQA-5 across multiple LALMs. Findings show strong answer accuracy does not guarantee reliable conversational repair, with repair emerging as the primary bottleneck and unanswerability recognition remaining difficult, especially for longer, acoustically complex inputs. The work advocates for evaluating and improving models’ conversational flexibility, treating unanswerability as a cue for explicit repair to better reflect real-world spoken interaction and reliability.

Abstract

Large Audio-Language Models (LALMs) have demonstrated strong performance in spoken question answering (QA), with existing evaluations primarily focusing on answer accuracy and robustness to acoustic perturbations. However, such evaluations implicitly assume that spoken inputs remain semantically answerable, an assumption that often fails in real-world interaction when essential information is missing. In this work, we introduce a repair-aware evaluation setting that explicitly distinguishes between answerable and unanswerable audio inputs. We define answerability as a property of the input itself and construct paired evaluation conditions using a semantic-acoustic masking protocol. Based on this setting, we propose the Evaluability Awareness and Repair (EAR) score, a non-compensatory metric that jointly evaluates task competence under answerable conditions and repair behavior under unanswerable conditions. Experiments on two spoken QA benchmarks across diverse LALMs reveal a consistent gap between answer accuracy and conversational reliability: while many models perform well when inputs are answerable, most fail to recognize semantic unanswerability and initiate appropriate conversational repair. These findings expose a limitation of prevailing accuracy-centric evaluation practices and motivate reliability assessments that treat unanswerable inputs as cues for repair and continued interaction.

Figures (5)

• Figure 1: Left: Traditional evaluation measures answer correctness under clean, answerable inputs. Middle: Robustness evaluation tests whether correct answers are maintained under acoustic perturbations that preserve answerability. Right: Our repair-aware evaluation masks answer-critical information to create unanswerable inputs and assesses whether models shift from answering to conversational repair.
• Figure 2: Overview of the repair-aware evaluation framework. The original audio generates two variants using the semantic-acoustic masking protocol: an answerable (semantic-invariant) and an unanswerable (semantic-degrading). The same set of LALMs then processes the original audio and its two masked variants. Finally, the LLM-as-a-judge evaluates these responses for answer correctness and conversational repair, respectively, under answerable and unanswerable inputs.
• Figure 3: Sensitivity of EAR to different semantic-degrading masking realizations on the WDYL dataset. EAR scores are reported under four masking types: white noise, silence, music, and multi-speaker. While absolute values vary, relative model ordering is preserved across masking realizations.
• Figure 4: Effects of prompting and mask severity on EAR scores on the WDYL dataset. We compare the base setting with a transcription-based prompt enhancement and an enhanced masking setting that expands the masked time window around the answer span. Increasing mask severity consistently improves EAR across most models, while prompt enhancement yields limited and model-dependent effects.
• Figure 5: Evaluation prompts designed for answerable and unanswerable inputs. Green blocks denote answerable evaluation prompts, while red blocks denote unanswerable evaluation prompts.