Semantic Proximity Alignment: Towards Human Perception-consistent Audio Tagging by Aligning with Label Text Description

TL;DR

The paper tackles the mismatch between one-hot supervision and semantic relationships in audio tagging by introducing Semantic Proximity Alignment ($SPA$), which uses label text descriptions as auxiliary supervision. It aligns audio features with text embeddings in a shared space using a cosine-based objective that combines BCE, ontology-aware BCE, and a text-audio alignment loss ($L_{SPA}$), with the text encoder kept frozen. Evaluations on Audioset show OmAP gains across five state-of-the-art encoders (approximately +$1.8$ OmAP) while mAP may decline, and human studies confirm better perceptual alignment under SPA. Overall, the work demonstrates that leveraging natural language descriptions of sound events can inject the semantic hierarchy and proximity into audio encoders, improving human-aligned tagging performance and offering a realistic path toward more robust audio understanding systems.

Abstract

Most audio tagging models are trained with one-hot labels as supervised information. However, one-hot labels treat all sound events equally, ignoring the semantic hierarchy and proximity relationships between sound events. In contrast, the event descriptions contains richer information, describing the distance between different sound events with semantic proximity. In this paper, we explore the impact of training audio tagging models with auxiliary text descriptions of sound events. By aligning the audio features with the text features of corresponding labels, we inject the hierarchy and proximity information of sound events into audio encoders, improving the performance while making the prediction more consistent with human perception. We refer to this approach as Semantic Proximity Alignment (SPA). We use Ontology-aware mean Average Precision (OmAP) as the main evaluation metric for the models. OmAP reweights the false positives based on Audioset ontology distance and is more consistent with human perception compared to mAP. Experimental results show that the audio tagging models trained with SPA achieve higher OmAP compared to models trained with one-hot labels solely (+1.8 OmAP). Human evaluations also demonstrate that the predictions of SPA models are more consistent with human perception.

Figures (3)

• Figure 1: t-SNE visualization of the text embeddings of the language description of 527 audio classes in Audioset evaluation subset. The method used to build the descriptions is Concat, as explained in Sec. \ref{['ssec:3_1']}
• Figure 2: Architecture of proposed Semantic Proximity Alignment (SPA). We employ a pretrained BERT devlin2018bert model as text encoder, which is kept frozen during training.
• Figure 3: Differences of OmAP at each coarse-grained level $\lambda$ of PaSST koutini2021efficient and PANN kong2020panns with or without SPA.