TSPE: Task-Specific Prompt Ensemble for Improved Zero-Shot Audio Classification

TL;DR

TSPE addresses zero-shot audio classification by replacing generic prompts with task-specific, hard prompts derived from label semantics and augmented via prompt ensembles. It uses GPT-4 to generate a tailored pool of attributes and sources per task, filters them for relevance, and averages their text embeddings to better align with audio representations, all without training. Across twelve datasets and two MS-CLAP ALMs, TSPE yields consistent absolute gains (1.23%–16.36%), though some cases show declines when linguistic richness is insufficient. The approach offers a practical, training-free path to boosting zero-shot performance and robustness to out-of-distribution data, with clear guidance for prompt generation and selection.

Abstract

Audio-language models (ALMs) excel in zero-shot audio classification, a task where models classify previously unseen audio clips at test time by leveraging descriptive natural language prompts. We introduce TSPE (Task-Specific Prompt Ensemble), a simple, training-free hard prompting method that boosts ALEs' zero-shot performance by customizing prompts for diverse audio classification tasks. Rather than using generic template-based prompts like "Sound of a car" we generate context-rich prompts, such as "Sound of a car coming from a tunnel". Specifically, we leverage label information to identify suitable sound attributes, such as "loud" and "feeble", and appropriate sound sources, such as "tunnel" and "street" and incorporate this information into the prompts used by Audio-Language Models (ALMs) for audio classification. Further, to enhance audio-text alignment, we perform prompt ensemble across TSPE-generated task-specific prompts. When evaluated on 12 diverse audio classification datasets, TSPE improves performance across ALMs by showing an absolute improvement of 1.23-16.36% over vanilla zero-shot evaluation.

Figures (2)

• Figure 1: Illustration of TSPE Workflow: We start with a pool of sound attributes and sources to customize a set of prompt templates using GPT-4. Then, we manually select prompts relevant to the specific downstream task and pass them to the text encoder to generate representations. These representations are averaged using a prompt ensemble. Finally, we compute the cosine similarity between the audio representation and the averaged text representation.
• Figure 2: Effect of different number of prompts on MSCLAP'23 for Audio Classification on the VocalSound dataset