SupCLAP: Controlling Optimization Trajectory Drift in Audio-Text Contrastive Learning with Support Vector Regularization

TL;DR

SupCLAP tackles optimization trajectory drift in contrastive audio-text pretraining by introducing Support Vector Regularization (SVR), which reshapes the gradient space to suppress the perpendicular component of pushing forces from negatives. SVR constructs a text support vector $t_{sup}=t^+ + R\hat{u}$ with a semantic radius $R$ modeled unsupervisedly via StaticSVR or DynamicSVR, and optimizes $L_{SupCLAP}=L_{orig}+\alpha L_{svr}$ to achieve a more direct convergence path. The work provides theoretical and empirical support that SVR accelerates convergence and improves alignment, with extensive experiments on AudioCaps and Clotho in both monolingual and multilingual settings, plus zero-shot classification benchmarks; results show significant gains over InfoNCE and SigLIP with negligible computational overhead. The dynamic radius predictor further enhances local control, adapting to hard negatives and dataset shifts, while constraint terms stabilize $R$ predictions. Overall, SVR offers a practical, data-efficient improvement for cross-modal audio-text representations, broadening the effectiveness of CLAP in real-world, multilingual contexts.

Abstract

Contrastive language-audio pretraining, which aims to unify multimodal representations in a shared embedding space, serves as a cornerstone for building a wide range of applications, from cross-modal retrieval to cutting-edge multimodal large language models. However, we find that the perpendicular component of the pushing force from negative samples in contrastive learning is a double-edged sword: it contains rich supplementary information from negative samples, yet its unconstrained nature causes optimization trajectory drift and training instability. To address this, we propose Support Vector Regularization (SVR), a method that introduces an auxiliary support vector to control this perpendicular component, aiming to harness its rich information while mitigating the associated trajectory drift. The efficacy of SVR is critically governed by its semantic radius, for which we explore two unsupervised modeling strategies: direct parameterization and an adaptive radius predictor module enhanced with constraints to improve its predicting accuracy. Extensive experimental results demonstrate that our method surpasses widely used baselines like InfoNCE and SigLIP loss across classification, monolingual retrieval, and multilingual retrieval on standard audio-text datasets. Both the theoretical analysis and the experimental results on optimizing trajectory drift validate the correctness and effectiveness of our SVR method. Notably, our method is highly efficient, it operates without the need for extra training data or inference computation, and adds only a negligible overhead to the training.

Figures (7)

• Figure 1: Optimization Trajectory Drift Analysis. Drift is measured by the cosine similarity between the update vector and the 'pulling force' vector; a higher similarity indicates lower drift. Compared to InfoNCE loss, our SVR method effectively mitigates this drift. This result confirms the existence of optimization trajectory drift.
• Figure 2: Illustration of Global and Local Perpendicular Components. The left subfigure depicts the global perpendicular component. The subfigure on the right illustrates the local perpendicular component. For clarity in demonstrating the local perpendicular component, the negative audio embeddings are shown in the right subfigure with distinct distributions across batches. In practice, the negative distributions across batches are more likely to overlap, as shown in the left subfigure.
• Figure 3: Results of Semantic Radius Changes
• Figure 4: Distribution of Positive Pair Similarity in AudioCaps
• Figure 5: Comparison of Convergence Speed between Baseline loss and SVR.