Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DiffATR: Diffusion-based Generative Modeling for Audio-Text Retrieval

Yifei Xin, Xuxin Cheng, Zhihong Zhu, Xusheng Yang, Yuexian Zou

TL;DR

This work presents a diffusion-based ATR framework (DiffATR), which models ATR as an iterative procedure that progressively generates joint distribution from noise, and consistently exhibits strong performance in out-of-domain retrieval settings.

Abstract

Existing audio-text retrieval (ATR) methods are essentially discriminative models that aim to maximize the conditional likelihood, represented as p(candidates|query). Nevertheless, this methodology fails to consider the intrinsic data distribution p(query), leading to difficulties in discerning out-of-distribution data. In this work, we attempt to tackle this constraint through a generative perspective and model the relationship between audio and text as their joint probability p(candidates,query). To this end, we present a diffusion-based ATR framework (DiffATR), which models ATR as an iterative procedure that progressively generates joint distribution from noise. Throughout its training phase, DiffATR is optimized from both generative and discriminative viewpoints: the generator is refined through a generation loss, while the feature extractor benefits from a contrastive loss, thus combining the merits of both methodologies. Experiments on the AudioCaps and Clotho datasets with superior performances, verify the effectiveness of our approach. Notably, without any alterations, our DiffATR consistently exhibits strong performance in out-of-domain retrieval settings.

DiffATR: Diffusion-based Generative Modeling for Audio-Text Retrieval

TL;DR

This work presents a diffusion-based ATR framework (DiffATR), which models ATR as an iterative procedure that progressively generates joint distribution from noise, and consistently exhibits strong performance in out-of-domain retrieval settings.

Abstract

Existing audio-text retrieval (ATR) methods are essentially discriminative models that aim to maximize the conditional likelihood, represented as p(candidates|query). Nevertheless, this methodology fails to consider the intrinsic data distribution p(query), leading to difficulties in discerning out-of-distribution data. In this work, we attempt to tackle this constraint through a generative perspective and model the relationship between audio and text as their joint probability p(candidates,query). To this end, we present a diffusion-based ATR framework (DiffATR), which models ATR as an iterative procedure that progressively generates joint distribution from noise. Throughout its training phase, DiffATR is optimized from both generative and discriminative viewpoints: the generator is refined through a generation loss, while the feature extractor benefits from a contrastive loss, thus combining the merits of both methodologies. Experiments on the AudioCaps and Clotho datasets with superior performances, verify the effectiveness of our approach. Notably, without any alterations, our DiffATR consistently exhibits strong performance in out-of-domain retrieval settings.
Paper Structure (14 sections, 8 equations, 3 figures, 7 tables)

This paper contains 14 sections, 8 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Existing Discriminative Modeling
  4. DiffATR: Generation Modeling
  5. Attention-based Denoising Module
  6. Generation-based Optimization
  7. Discrimination-based Optimization
  8. Experiments
  9. Datasets
  10. Training Details and Evaluation metrics
  11. Experimental Results
  12. Ablation Study
  13. Visualization
  14. Conclusions

Figures (3)

  • Figure 1: (a) The diffusion model for joint probability generation. (b) Overview of our DiffATR framework.
  • Figure 2: Model Architecture of the Attention-based Denosing Module.
  • Figure 3: The visualization of the diffusion process of the probability distribution.