Generalizable Prompt Tuning for Audio-Language Models via Semantic Expansion

TL;DR

This work tackles the generalization gap in prompt tuning for Audio-Language Models caused by the Base-New Tradeoff and degraded semantic structure in the prompt embedding space. It introduces Semantically Expanded Prompt Tuning (SEPT), a plug-and-play framework that uses large language model–generated semantic neighbors and a margin-based intra-class/inter-class loss to form cohesive, discriminative class clusters in text space. The authors establish the first ALM generalization benchmark covering base-to-new and cross-dataset transfer, and demonstrate that SEPT consistently improves generalization across multiple prompt-tuning baselines without adding inference cost. Key contributions include the semantic expansion loss, margin definitions m_{i,i,n} and m_{i,j,n} computed over $T$ prompts, and extensive empirical validation showing improved base-new performance and transferability in diverse audio tasks.

Abstract

Prompt tuning has achieved remarkable progress in vision-language models (VLMs) and is recently being adopted for audio-language models (ALMs). However, its generalization ability in ALMs remains largely underexplored. We observe that conventional prompt tuning for ALMs also suffers from the Base-New Tradeoff, and we identify that this issue stems from the disrupted semantic structure of the embedding space. To address this issue, we propose Semantically Expanded Prompt Tuning (SEPT)-a plug-and-play framework that explicitly regularizes the prompt embedding space by incorporating semantic neighbors generated by large language models. SEPT introduces a novel semantic expansion loss with margin constraints that promote intra-class compactness and inter-class separability, thereby enhancing the semantic structure of the prompt embedding space. For comprehensive evaluation, we establish the first benchmark setup for prompt generalization in ALMs, covering both base-to-new generalization and cross-dataset transferability. Extensive experiments demonstrate that SEPT consistently improves generalization performance across multiple prompt tuning baselines, while maintaining computational cost during inference. Codes are available in https://github.com/jhyukjang/SEPT.

Figures (8)

• Figure 1: Motivation. The $y$-axis corresponds to class names, and $x$-axis lists semantic neighbors for each class. Hand-crafted zero-shot prompts show strong similarity between each class and its neighbors. In contrast, conventional prompt tuning disrupts this alignment. SEPT preserves this semantic similarity, keeping strong alignment between each class and its neighbors.
• Figure 2: Framework of proposed Semantically Expanded Prompt Tuning (SEPT), applied to CoOp coop. During training, we use LLM to generate Semantic Neighbors-words that are semantically related to each class name. These neighbors are encoded into semantic neighbor embeddings using the same learnable prompt and frozen text encoder. To preserve the semantic structure of the original text embedding space, we introduce a semantic expansion loss $\mathcal{L}_{\text{se}}$. Specifically, as illustrated in the figure, $\mathcal{L}_{\text{intra}}$ pulls class embeddings closer to far semantic neighbors, and $\mathcal{L}_{\text{inter}}$ pushes away from overly close unrelated ones. Note that our method can be integrated into other prompt tuning methods and does not affect inference efficiency.
• Figure 3: Ablation studies of the hyperparameter $\lambda$ (left) and the number of semantic neighbors $N$ (right).
• Figure 4: t-SNE visualization of prompt embeddings of new classes and their semantic neighbors from ESC50-Actions. $\diamond$ denotes the prompt embedding of a new class, while $\circ$ represents its corresponding semantic neighbors.
• Figure 5: Prompt for diverse prefixes.