PAT: Parameter-Free Audio-Text Aligner to Boost Zero-Shot Audio Classification

TL;DR

This paper introduces PAT (Parameter-free Audio-Text aligner), a simple and training-free method aimed at boosting the zero-shot audio classification performance of CLAP-like ALMs and demonstrates that PAT maintains robust performance even when input audio is degraded by varying levels of noise.

Abstract

Audio-Language Models (ALMs) have demonstrated remarkable performance in zero-shot audio classification. In this paper, we introduce PAT (Parameter-free Audio-Text aligner), a simple and training-free method aimed at boosting the zero-shot audio classification performance of CLAP-like ALMs. To achieve this, we propose to improve the cross-modal interaction between audio and language modalities by enhancing the representations for both modalities using mutual feedback. Precisely, to enhance textual representations, we propose a prompt ensemble algorithm that automatically selects and combines the most relevant prompts from a datastore with a large pool of handcrafted prompts and weighs them according to their relevance to the audio. On the other hand, to enhance audio representations, we reweigh the frame-level audio features based on the enhanced textual information. Our proposed method does not require any additional modules or parameters and can be used with any existing CLAP-like ALM to improve zero-shot audio classification performance. We experiment across 18 diverse benchmark datasets and 6 ALMs and show that the PAT outperforms vanilla zero-shot evaluation with significant margins of 0.42%-27.0%. Additionally, we demonstrate that PAT maintains robust performance even when input audio is degraded by varying levels of noise. Our code will be open-sourced upon acceptance.

Figures (4)

• Figure 1: Comparison of zero-shot audio classification performance of ALEs (LAION CLAP laionclap2023 and MS CLAP elizalde2023clap) with and without PaT on MUSDB Bertin-Mahieux2011 (music genre classification). Our proposed training-free method significantly enhances zero-shot performance, even in low-resource domains where the ALEs have limited training data.
• Figure 2: Illustration of PaT. PaT improves the zero-shot capabilities of ALEs by enriching audio-text representations in a parameter and training-free fashion. PaT consists of two major components: ① Weighted prompt ensemble that first utilizes an in-house generic prompt datastore to transform class labels into diverse textual descriptions, which are then encoded by a text encoder. Further, each prompt is assigned a unique score based on the level of uncertainty it introduces during zero-shot prediction (less uncertainty results in a higher score). A weighted average is then performed to generate task-specific, semantically rich textual representations. ② Next, the enriched textual representations are used to guide the enhancement of audio representations using a novel zero-shot cross model alignment. Precisely, frame-level audio representations are paired with enhanced textual representations to compute a parameter-free attention map, which is used in performing audio and text-guided transformations. Finally, the transformed frame-level audio representations are pooled, and the audio-text-guided information is added to the original logit space, boosting the ALE's zero-shot prediction capabilities.
• Figure 3: Examples of prompt templates and their paraphrased versions used in the prompt datastore.
• Figure 4: Measuring MSCLAP-23 zero-shot performance with the prompt ensemble (PE) vs. PaT on NSynth-Src by varying the prompt count. Generally, PaT outperforms PE across different prompt counts.