MAEB: Massive Audio Embedding Benchmark

TL;DR

MAEB addresses the need for a unified, large-scale evaluation of audio embeddings by extending the MTEB framework to audio with 30 tasks across 100+ languages. It introduces principled task and dataset selection, a Borda-based ranking scheme, and six task types (classification, zero-shot classification, clustering, retrieval, pair classification, reranking) to enable comprehensive, scalable evaluation across speech, music, environmental sounds, and cross-modal audio-text reasoning. Key findings show there is no universal audio encoder; multilingual understanding remains challenging, with clear trade-offs between acoustic and linguistic representations, and clustering remains a tough, underexplored area. MAEB correlates with downstream Audio LLM performance, supporting its utility for multimodal audio understanding; the benchmark is integrated into the MTEB ecosystem and released with MAEB+ tasks and tools to advance community-driven progress.

Abstract

We introduce the Massive Audio Embedding Benchmark (MAEB), a large-scale benchmark covering 30 tasks across speech, music, environmental sounds, and cross-modal audio-text reasoning in 100+ languages. We evaluate 50+ models and find that no single model dominates across all tasks: contrastive audio-text models excel at environmental sound classification (e.g., ESC50) but score near random on multilingual speech tasks (e.g., SIB-FLEURS), while speech-pretrained models show the opposite pattern. Clustering remains challenging for all models, with even the best-performing model achieving only modest results. We observe that models excelling on acoustic understanding often perform poorly on linguistic tasks, and vice versa. We also show that the performance of audio encoders on MAEB correlates highly with their performance when used in audio large language models. MAEB is derived from MAEB+, a collection of 98 tasks. MAEB is designed to maintain task diversity while reducing evaluation cost, and it integrates into the MTEB ecosystem for unified evaluation across text, image, and audio modalities. We release MAEB and all 98 tasks along with code and a leaderboard at https://github.com/embeddings-benchmark/mteb.

Figures (5)

• Figure 1: overview of task types and example subtypes in MAEB+. Values in parentheses denote numbers for MAEB.
• Figure 2: Domain-level performance on 94 tasks in MAEB+. Radial plot shows the top-performing model for each of the five acoustic domains: Speech (44 tasks), Music (13), Environmental (29), Bioacoustics (2), and Emotion (6). The dashed line represents an 80 target for universal performance, which remains unmet. Scores are averaged across all available task types (classification, clustering, retrieval, reranking). See Appendix \ref{['app:radar_methodology']} for methodology.
• Figure 3: MAEB+ embedding quality correlates with Audio LLM performance. MMAU evaluates Audio LLMs across Speech, Music, and Sound, the same domains covered by MAEB+. Each point plots an Audio LLM's overall MMAU score (y-axis, averaged across domains) against its encoder's MAEB+ score (x-axis, computed from 26 classification tasks aligned with MMAU domains). Preliminary correlation (R²=0.86, p=0.072, n=4) suggests a positive relationship between embedding quality and downstream reasoning, though the small sample size and statistical marginality warrant caution in interpreting this relationship.
• Figure 4: Language distribution in the MAEB+ collection. English dominates with 70 tasks. We use zxx (No Linguistic Content) to tag datasets with no languages present.
• Figure 5: Domain distributions in the MAEB+ collection, MAEB, and MAEB(audio-only).