Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The CMU-AIST submission for the ICME 2025 Audio Encoder Challenge

Shikhar Bharadwaj, Samuele Cornell, Kwanghee Choi, Hye-jin Shim, Soham Deshmukh, Satoru Fukayama, Shinji Watanabe

TL;DR

The paper tackles improving self-supervised, transformer-based audio encoders by expanding BEATs pre-training to 74k hours across multiple domains and evaluating how data mix (speech-heavy vs balanced) affects domain-specific performance. It introduces a simple three-model ensemble that fuses two scaled-up BEATs (300M) trained on different mixtures with the Dasheng 1.2B model via upsampling and feature concatenation, achieving robust performance across benchmarks. Key findings show data-domain composition substantially shapes SSL encoder effectiveness, and the ensemble can surpass individual models, highlighting the value of domain-aware pre-training and simple fusion strategies. The work provides open-source checkpoints and suggests mixture-of-experts as a promising direction for future audio encoding architectures.

Abstract

This technical report describes our submission to the ICME 2025 audio encoder challenge. Our submitted system is built on BEATs, a masked speech token prediction based audio encoder. We extend the BEATs model using 74,000 hours of data derived from various speech, music, and sound corpora and scale its architecture upto 300 million parameters. We experiment with speech-heavy and balanced pre-training mixtures to study the impact of different domains on final performance. Our submitted system consists of an ensemble of the Dasheng 1.2 billion model with two custom scaled-up BEATs models trained on the aforementioned pre-training data mixtures. We also propose a simple ensembling technique that retains the best capabilities of constituent models and surpasses both the baseline and Dasheng 1.2B. For open science, we publicly release our trained checkpoints via huggingface at https://huggingface.co/shikhar7ssu/OpenBEATs-ICME-SOUND and https://huggingface.co/shikhar7ssu/OpenBEATs-ICME.

The CMU-AIST submission for the ICME 2025 Audio Encoder Challenge

TL;DR

The paper tackles improving self-supervised, transformer-based audio encoders by expanding BEATs pre-training to 74k hours across multiple domains and evaluating how data mix (speech-heavy vs balanced) affects domain-specific performance. It introduces a simple three-model ensemble that fuses two scaled-up BEATs (300M) trained on different mixtures with the Dasheng 1.2B model via upsampling and feature concatenation, achieving robust performance across benchmarks. Key findings show data-domain composition substantially shapes SSL encoder effectiveness, and the ensemble can surpass individual models, highlighting the value of domain-aware pre-training and simple fusion strategies. The work provides open-source checkpoints and suggests mixture-of-experts as a promising direction for future audio encoding architectures.

Abstract

This technical report describes our submission to the ICME 2025 audio encoder challenge. Our submitted system is built on BEATs, a masked speech token prediction based audio encoder. We extend the BEATs model using 74,000 hours of data derived from various speech, music, and sound corpora and scale its architecture upto 300 million parameters. We experiment with speech-heavy and balanced pre-training mixtures to study the impact of different domains on final performance. Our submitted system consists of an ensemble of the Dasheng 1.2 billion model with two custom scaled-up BEATs models trained on the aforementioned pre-training data mixtures. We also propose a simple ensembling technique that retains the best capabilities of constituent models and surpasses both the baseline and Dasheng 1.2B. For open science, we publicly release our trained checkpoints via huggingface at https://huggingface.co/shikhar7ssu/OpenBEATs-ICME-SOUND and https://huggingface.co/shikhar7ssu/OpenBEATs-ICME.
Paper Structure (8 sections, 1 figure, 2 tables)

This paper contains 8 sections, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. System Description
  3. Background
  4. Method
  5. Ensembling Technique
  6. Pre-training Datasets
  7. Results
  8. Conclusion

Figures (1)

  • Figure 1: Block diagram of the submitted system: we concatenate the output embedding feature vectors from three different systems. Two are based on the BEATs framework and one is the pre-trained Dasheng 1.2B model. We upsample the Dasheng embedding sequence to match the count of embeddings from the BEATs models and concatenate the outputs of the three models. The BEATs models are scaled up to 300M parameter and trained on different pre-training mixtures (speech heavy and balanced) from a 74 k hour data pool.