SmoothCLAP: Soft-Target Enhanced Contrastive Language\--Audio Pretraining for Affective Computing

TL;DR

The paper addresses the limitation of CLAP's strict one-to-one audio–text alignment by exploiting intra-modal relationships to provide soft supervision for affective representations. SmoothCLAP constructs soft targets from audio-to-audio and text-to-text similarities using computational paralinguistics and combines them with traditional one-hot supervision via mix parameters $\gamma$ and $\beta$, optimized with a symmetric KL loss against targets $y_{ij}$ where $y_{ij}=(1-\beta)\delta_{ij}+\beta q_{ij}$ and $q_{ij}=(1-\gamma)q^{a2a}_{ij}+\gamma q^{t2t}_{ij}$. Evaluated on MSP-Podcast v1.9 plus eight test sets in English and German, SmoothCLAP yields improvements in unweighted average recall (UAR) on 5 of 8 tasks and exhibits strong zero-shot cross-lingual transfer. These findings demonstrate that incorporating paralinguistic priors into contrastive language-audio pretraining yields more nuanced embeddings and suggests broader applicability beyond emotion labels.

Abstract

The ambiguity of human emotions poses several challenges for machine learning models, as they often overlap and lack clear delineating boundaries. Contrastive language-audio pretraining (CLAP) has emerged as a key technique for generalisable emotion recognition. However, as conventional CLAP enforces a strict one-to-one alignment between paired audio-text samples, it overlooks intra-modal similarity and treats all non-matching pairs as equally negative. This conflicts with the fuzzy boundaries between different emotions. To address this limitation, we propose SmoothCLAP, which introduces softened targets derived from intra-modal similarity and paralinguistic features. By combining these softened targets with conventional contrastive supervision, SmoothCLAP learns embeddings that respect graded emotional relationships, while retaining the same inference pipeline as CLAP. Experiments on eight affective computing tasks across English and German demonstrate that SmoothCLAP is consistently achieving superior performance. Our results highlight that leveraging soft supervision is a promising strategy for building emotion-aware audio-text models.

Figures (3)

• Figure 1: Architecture of SmoothCLAP. The model integrates intra-modal self-similarity to enable soft-target smoothing. It promotes alignment across modalities while preserving semantic relationships within both the audio and text domains.
• Figure 2: Confusion matrices on IEMOCAP for ParaCLAP (left) and SmoothCLAP (right).
• Figure 3: Performance of mix gamma $\gamma$ and fusion factor $\beta$