Which Evaluation for Which Model? A Taxonomy for Speech Model Assessment
Maureen de Seyssel, Eeshan Gunesh Dhekane
TL;DR
The paper tackles fragmentation in evaluating speech foundation models by introducing a capability-aware taxonomy with three orthogonal axes: evaluation aspect, foundation model capability, and task/protocol requirements. It maps existing benchmarks to these axes to reveal coverage gaps and guide principled evaluation design, including representations, discrete units, probabilistic scoring, and generation. The framework is intended as a practical decision tool to match model interfaces with appropriate evaluations and to highlight missing benchmarks in areas like prosody, interaction, and cross-modal reasoning. By promoting a unified language for model–task alignment, the work aims to improve interpretability, comparability, and the design of future benchmarks for speech models.
Abstract
Speech foundation models have recently achieved remarkable capabilities across a wide range of tasks. However, their evaluation remains disjointed across tasks and model types. Different models excel at distinct aspects of speech processing and thus require different evaluation protocols. This paper proposes a unified taxonomy that addresses the question: Which evaluation is appropriate for which model? The taxonomy defines three orthogonal axes: the evaluation aspect being measured, the model capabilities required to attempt the task, and the task or protocol requirements needed to perform it. We classify a broad set of existing evaluations and benchmarks along these axes, spanning areas such as representation learning, speech generation, and interactive dialogue. By mapping each evaluation to the capabilities a model exposes (e.g., speech generation, real-time processing) and to its methodological demands (e.g., fine-tuning data, human judgment), the taxonomy provides a principled framework for aligning models with suitable evaluation methods. It also reveals systematic gaps, such as limited coverage of prosody, interaction, or reasoning, that highlight priorities for future benchmark design. Overall, this work offers a conceptual foundation and practical guide for selecting, interpreting, and extending evaluations of speech models.