Robustness assessment of large audio language models in multiple-choice evaluation
Fernando López, Santosh Kesiraju, Jordi Luque
TL;DR
This work tackles the reliability of MCQA-based evaluation for large audio language models (LALMs) by exposing their sensitivity to subtle text perturbations in options, questions, and distractors. It introduces a robust perturbation framework that applies isolated and mixed perturbations to questions, ground-truth answers, and distractors, and uses Consistency Rate ($CR$) and Correctness Rate ($CoR$) to quantify internal and external stability, respectively: $CR = \frac{1}{n} \sum_{i=1}^{n} \sum_{j=1}^{m} \sum_{k=j+1}^{m} \frac{\delta(r_{ij}, r_{ik})}{\binom{m}{2}}$ and $CoR = \frac{1}{n} \sum_{i=1}^{n} \frac{1}{m} \sum_{j=1}^{m} correctness(r_{ij}, g_i)$. Through experiments on MMAU, MMAR, and MMSU with four LALMs, the study finds that distractor rephrasing yields the largest accuracy drops and reveals a bias toward longer choices, with Audio Flamingo 3 exhibiting the strongest overall robustness and Qwen-Omni showing solid stability to distractor wording. The proposed mix-perturbation protocol enables more trustworthy benchmarking by capturing variability beyond mean accuracy, highlighting the need for robustness-aware metrics in LALM MCQA evaluation.
Abstract
Recent advances in large audio language models (LALMs) have primarily been assessed using a multiple-choice question answering (MCQA) framework. However, subtle changes, such as shifting the order of choices, result in substantially different results. Existing MCQA frameworks do not account for this variability and report a single accuracy number per benchmark or category. We dive into the MCQA evaluation framework and conduct a systematic study spanning three benchmarks (MMAU, MMAR and MMSU) and four models: Audio Flamingo 2, Audio Flamingo 3, Qwen2.5-Omni-7B-Instruct, and Kimi-Audio-7B-Instruct. Our findings indicate that models are sensitive not only to the ordering of choices, but also to the paraphrasing of the question and the choices. Finally, we propose a simpler evaluation protocol and metric that account for subtle variations and provide a more detailed evaluation report of LALMs within the MCQA framework.