The ICME 2025 Audio Encoder Capability Challenge

TL;DR

The ICME 2025 Audio Encoder Capability Challenge targets robust evaluation of continuous audio encoders across speech, environmental sounds, and music using two evaluation tracks: Track A (linear fine-tuning) and Track B (unparameterized KNN evaluation). It employs a diverse, open dataset ecosystem with both public data and hidden organizer datasets, and requires a single encoder to emit frame-level embeddings and an utterance embedding via a defined API. Normalized metrics and a weighted scoring scheme enable fair cross-task comparisons, with separate leaderboards for parameterized and unparameterized evaluations. This work extends prior benchmarks by emphasizing continuous representations and real-world applicability, providing an open evaluation platform to drive advances in audio representations for multimodal and industrial AI systems.

Abstract

This challenge aims to evaluate the capabilities of audio encoders, especially in the context of multi-task learning and real-world applications. Participants are invited to submit pre-trained audio encoders that map raw waveforms to continuous embeddings. These encoders will be tested across diverse tasks including speech, environmental sounds, and music, with a focus on real-world usability. The challenge features two tracks: Track A for parameterized evaluation, and Track B for parameter-free evaluation. This challenge provides a platform for evaluating and advancing the state-of-the-art in audio encoder design.