Leveraging Prediction Entropy for Automatic Prompt Weighting in Zero-Shot Audio-Language Classification

TL;DR

The paper tackles prompt sensitivity in zero-shot audio-language classification by introducing an entropy-minimization framework to weight a set of text templates. It learns a template weight vector beta on the simplex to minimize prediction entropy, combining template logits via bar_l and deriving class probabilities p through a temperature-controlled softmax: p_i_k = exp(bar_l_ik / tau) / sum_l exp(bar_l_il / tau). The objective blends prediction entropy, a zero-shot regularization term, and an entropy penalty on beta, and beta is optimized with a fixed-point update, after precomputing embeddings with CLAP-2022. Across five audio benchmarks and 35 prompts, the method yields consistent gains over baselines, including pruning-based and average embedding ensembles, with notable improvements on challenging datasets, while maintaining negligible runtime overhead. This approach offers a practical, label-free enhancement to ALM zero-shot pipelines by robustly fusing prompts without resorting to additional labeled data.

Abstract

Audio-language models have recently demonstrated strong zero-shot capabilities by leveraging natural-language supervision to classify audio events without labeled training data. Yet, their performance is highly sensitive to the wording of text prompts, with small variations leading to large fluctuations in accuracy. Prior work has mitigated this issue through prompt learning or prompt ensembling. However, these strategies either require annotated data or fail to account for the fact that some prompts may negatively impact performance. In this work, we present an entropy-guided prompt weighting approach that aims to find a robust combination of prompt contributions to maximize prediction confidence. To this end, we formulate a tailored objective function that minimizes prediction entropy to yield new prompt weights, utilizing low-entropy as a proxy for high confidence. Our approach can be applied to individual samples or a batch of audio samples, requiring no additional labels and incurring negligible computational overhead. Experiments on five audio classification datasets covering environmental, urban, and vocal sounds, demonstrate consistent gains compared to classical prompt ensembling methods in a zero-shot setting, with accuracy improvements 5-times larger across the whole benchmark.

Figures (1)

• Figure 1: Summary of classification accuracy improvement over zero-shot prediction. Our approach, applied on the entirety of each dataset, shows consistent improvement over majority voting and embedding averaging ensemble methods.