Do Audio-Language Models Understand Linguistic Variations?
Ramaneswaran Selvakumar, Sonal Kumar, Hemant Kumar Giri, Nishit Anand, Ashish Seth, Sreyan Ghosh, Dinesh Manocha
TL;DR
This work reveals that open-vocabulary audio-language models (ALMs) struggle to generalize across linguistic variations in textual queries, with notable drops in text-to-audio retrieval. It introduces RobustCLAP, a compute-efficient method that uses a multi-view contrastive objective to align paraphrase views with the original captions, enabling robustness to paraphrase-based query changes while preserving pretraining knowledge. The authors construct five paraphrased benchmarks and demonstrate that RobustCLAP improves retrieval on original benchmarks by up to 13% and mitigates paraphrase-induced performance losses by up to 12%, with negligible impact on zero-shot audio classification. The approach highlights a practical path toward more reliable, linguistically flexible audio search, and invites further exploration of paraphrase-driven robustness in ALMs.
Abstract
Open-vocabulary audio language models (ALMs), like Contrastive Language Audio Pretraining (CLAP), represent a promising new paradigm for audio-text retrieval using natural language queries. In this paper, for the first time, we perform controlled experiments on various benchmarks to show that existing ALMs struggle to generalize to linguistic variations in textual queries. To address this issue, we propose RobustCLAP, a novel and compute-efficient technique to learn audio-language representations agnostic to linguistic variations. Specifically, we reformulate the contrastive loss used in CLAP architectures by introducing a multi-view contrastive learning objective, where paraphrases are treated as different views of the same audio scene and use this for training. Our proposed approach improves the text-to-audio retrieval performance of CLAP by 0.8%-13% across benchmarks and enhances robustness to linguistic variation.