Diverse Audio Embeddings -- Bringing Features Back Outperforms CLAP!
Prateek Verma
TL;DR
The paper tackles the dominance of end-to-end audio models by proposing a multi-embedding framework that combines domain-specific handcrafted features (pitch/frequency content and timbre) with end-to-end and Neuralogram-based representations. Each representation is mapped to a fixed 64-dim embedding and processed by a Transformer, with separate training and a final linear classifier that blends the diverse embeddings. Empirical results on FSD50K show that while individual streams vary in strength, the combined diverse-embedding approach achieves a substantial MAP of 59.6, surpassing CLAP and PLSA without using extra training data and with a compact model (~4M parameters). This demonstrates that integrating interpretable, domain-informed features with modern end-to-end architectures can yield robust, high-performance audio representations and suggests a path for feature engineering to complement large-scale learning.
Abstract
With the advent of modern AI architectures, a shift has happened towards end-to-end architectures. This pivot has led to neural architectures being trained without domain-specific biases/knowledge, optimized according to the task. We in this paper, learn audio embeddings via diverse feature representations, in this case, domain-specific. For the case of audio classification over hundreds of categories of sound, we learn robust separate embeddings for diverse audio properties such as pitch, timbre, and neural representation, along with also learning it via an end-to-end architecture. We observe handcrafted embeddings, e.g., pitch and timbre-based, although on their own, are not able to beat a fully end-to-end representation, yet adding these together with end-to-end embedding helps us, significantly improve performance. This work would pave the way to bring some domain expertise with end-to-end models to learn robust, diverse representations, surpassing the performance of just training end-to-end models.