Mind the Gap! Static and Interactive Evaluations of Large Audio Models

TL;DR

This study investigates how to align Large Audio Models (LAMs) with user needs by contrasting interactive, user-driven evaluations against traditional static benchmarks. It collects 7,500 interactions from 484 participants across six LAM configurations to identify real-use tasks and user preferences, revealing that no single static benchmark reliably predicts interactive performance ($\tau \leq 0.33$) and that aggregated proxies explain only about $R^2=0.30$. The findings show that interactive preference often favors an ASR+LLM pipeline (Whisper+Llama) over top static performers, emphasizing that user efficiency-focused tasks (knowledge expansion, task execution) drive satisfaction. The work highlights a gap between static evaluation and real user experiences, arguing for new audio-centric benchmarks and interactive evaluation frameworks to guide development and benchmarking of LAMs in voice-enabled applications.

Abstract

As AI chatbots become ubiquitous, voice interaction presents a compelling way to enable rapid, high-bandwidth communication for both semantic and social signals. This has driven research into Large Audio Models (LAMs) to power voice-native experiences. However, aligning LAM development with user goals requires a clear understanding of user needs and preferences to establish reliable progress metrics. This study addresses these challenges by introducing an interactive approach to evaluate LAMs and collecting 7,500 LAM interactions from 484 participants. Through topic modeling of user queries, we identify primary use cases for audio interfaces. We then analyze user preference rankings and qualitative feedback to determine which models best align with user needs. Finally, we evaluate how static benchmarks predict interactive performance - our analysis reveals no individual benchmark strongly correlates with interactive results ($τ\leq 0.33$ for all benchmarks). While combining multiple coarse-grained features yields modest predictive power ($R^2$=$0.30$), only two out of twenty datasets on spoken question answering and age prediction show significantly positive correlations. This suggests a clear need to develop LAM evaluations that better correlate with user preferences.

Figures (8)

• Figure 1: Comparison of static and interactive ways of evaluating Large Audio Models. In this work, we perform interactive evaluations to understand how LAMs are likely to be used and how they can be benchmarked.
• Figure 2: We identify four main topics in user queries --- task execution, knowledge expansion, chat, advice seeking as well as sub-topics under each category through hierarchical clustering (left) and analyze the relative proportions of each query type (right).
• Figure 3: Head-to-head model comparisons (left) and Bradley-Terry (right) Scores from our evaluation. For win rates, * indicates the difference between preferences is significant (P<0.05) by a pairwise bootstrap test. For Bradley-Terry scores, distributions are shown shown for 10,000 bootstraps. $\dagger$ denotes an ASR + LLM pipeline.
• Figure 4: Mixed-effect regression of benchmark performance differences as a predictor of user preferences across models. 15 other features were pre-screened using VIF thresholding (threshold=10.0) with ties removed. Model fitting was performed fixed effects for benchmarks and random effects for model identity. The model achieved conditional/marginal $R^2$ of 0.99/0.30.
• Figure A.1: PCA Analysis of model performance on 20 static benchmarks.