Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ProLAP: Probabilistic Language-Audio Pre-Training

Toranosuke Manabe, Yuchi Ishikawa, Hokuto Munakata, Tatsuya Komatsu

TL;DR

ProLAP tackles the challenge of many-to-many, uncertain relationships between language and audio by modeling each input as a Gaussian distribution in a shared embedding space, extending CLAP with probabilistic representations. It introduces hierarchical inclusion loss and mask repulsive loss to explicitly capture semantic hierarchies within and across modalities, enabling more data-efficient learning even on smaller datasets. Empirically, ProLAP outperforms deterministic baselines on audio–text retrieval benchmarks and exhibits stronger hierarchical understanding via the proposed audio traversal analysis. The work advances open-vocabulary audio understanding by combining probabilistic embeddings with structured masking to reveal nuanced semantic structure and uncertainty in multimodal data.

Abstract

Language-audio joint representation learning frameworks typically depend on deterministic embeddings, assuming a one-to-one correspondence between audio and text. In real-world settings, however, the language-audio relationship is inherently many-to-many: one audio segment can be described by multiple captions and vice versa. To address this, we propose Probabilistic Language-Audio Pre-training (ProLAP), which models multiplicity as the spread of probability distributions in a joint language-audio embedding space. To train the intra-modal hierarchical relationship effectively, we also introduce two objectives: (i) hierarchical inclusion loss to promote semantic hierarchical understanding of inputs and (ii) mask repulsive loss to improve the efficiency of learning when optimizing the hierarchical inclusion loss. With this training strategy, our model can learn the hierarchical structure inherent in the data even from small datasets, in contrast to prior probabilistic approaches that rely on large-scale datasets. In our experiments, ProLAP outperforms existing deterministic approaches on audio-text retrieval tasks. Moreover, through experiments on the audio traversal task introduced in this paper, we demonstrate that ProLAP captures the plausible semantic hierarchy.

ProLAP: Probabilistic Language-Audio Pre-Training

TL;DR

ProLAP tackles the challenge of many-to-many, uncertain relationships between language and audio by modeling each input as a Gaussian distribution in a shared embedding space, extending CLAP with probabilistic representations. It introduces hierarchical inclusion loss and mask repulsive loss to explicitly capture semantic hierarchies within and across modalities, enabling more data-efficient learning even on smaller datasets. Empirically, ProLAP outperforms deterministic baselines on audio–text retrieval benchmarks and exhibits stronger hierarchical understanding via the proposed audio traversal analysis. The work advances open-vocabulary audio understanding by combining probabilistic embeddings with structured masking to reveal nuanced semantic structure and uncertainty in multimodal data.

Abstract

Language-audio joint representation learning frameworks typically depend on deterministic embeddings, assuming a one-to-one correspondence between audio and text. In real-world settings, however, the language-audio relationship is inherently many-to-many: one audio segment can be described by multiple captions and vice versa. To address this, we propose Probabilistic Language-Audio Pre-training (ProLAP), which models multiplicity as the spread of probability distributions in a joint language-audio embedding space. To train the intra-modal hierarchical relationship effectively, we also introduce two objectives: (i) hierarchical inclusion loss to promote semantic hierarchical understanding of inputs and (ii) mask repulsive loss to improve the efficiency of learning when optimizing the hierarchical inclusion loss. With this training strategy, our model can learn the hierarchical structure inherent in the data even from small datasets, in contrast to prior probabilistic approaches that rely on large-scale datasets. In our experiments, ProLAP outperforms existing deterministic approaches on audio-text retrieval tasks. Moreover, through experiments on the audio traversal task introduced in this paper, we demonstrate that ProLAP captures the plausible semantic hierarchy.
Paper Structure (16 sections, 9 equations, 3 figures, 3 tables)

This paper contains 16 sections, 9 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Model Overview
  4. ProLAP Baseline Objective
  5. Probabilistic Pairwise Contrastive Loss
  6. Inclusion Loss
  7. Hierarchical Understanding
  8. Hierarchical Inclusion Loss
  9. Mask Repulsive Loss
  10. Experiments
  11. Experimental Setup
  12. Comparison on AudioCaps and ClothoV2
  13. Understanding Learned Uncertainty
  14. Audio Traversal
  15. Ablation Studies
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Overview of ProLAP. ProLAP models the representation as a Gaussian random variable following $\mathcal{N}(\mu, \sigma^2)$. Audio-text alignment is learned using PPCL (Eq. \ref{['eq:PPCL']}), while semantic hierarchical understanding is encouraged by inclusion loss (Eqs. \ref{['eq:inc']} and \ref{['eq:h-inc']}). Mask repulsive loss (Eq. \ref{['eq:MR']}) prevents degenerate of representations for masked inputs.
  • Figure 2: Text length vs uncertainty.
  • Figure 3: Visualization of probabilistic audio embeddings.