Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FineLAP: Taming Heterogeneous Supervision for Fine-grained Language-Audio Pretraining

Xiquan Li, Xuenan Xu, Ziyang Ma, Wenxi Chen, Haolin He, Qiuqiang Kong, Xie Chen

Abstract

Contrastively pretrained audio-language models (e.g., CLAP) excel at clip-level understanding but struggle with frame-level tasks. Existing extensions fail to exploit the varying granularity of real-world audio-text data, where massive clip-level textual descriptions coexist with limited frame-level annotations. This paper proposes Fine-grained Language-Audio Pretraining (FineLAP), a novel training paradigm that advances both clip- and frame-level alignment in CLAP with heterogeneous data. FineLAP introduces a dual-stream sigmoid loss with a cluster-based sampling strategy to jointly learn from clip- and frame-level supervision. To capture both global semantics and local details, FineLAP uses a decoupled audio projector on top of a self-supervised encoder. To alleviate the scarcity of temporally annotated data, we present FineLAP-100k, a large-scale synthetic SED dataset constructed through a scalable curation pipeline. Extensive experiments demonstrate that FineLAP achieves SOTA performance across multiple audio understanding tasks, including retrieval, classification, sound event detection, and text-to-audio grounding. Ablation studies further show that coarse- and fine-grained alignment are mutually beneficial, providing insights for building better audio-language models (ALMs).

FineLAP: Taming Heterogeneous Supervision for Fine-grained Language-Audio Pretraining

Abstract

Contrastively pretrained audio-language models (e.g., CLAP) excel at clip-level understanding but struggle with frame-level tasks. Existing extensions fail to exploit the varying granularity of real-world audio-text data, where massive clip-level textual descriptions coexist with limited frame-level annotations. This paper proposes Fine-grained Language-Audio Pretraining (FineLAP), a novel training paradigm that advances both clip- and frame-level alignment in CLAP with heterogeneous data. FineLAP introduces a dual-stream sigmoid loss with a cluster-based sampling strategy to jointly learn from clip- and frame-level supervision. To capture both global semantics and local details, FineLAP uses a decoupled audio projector on top of a self-supervised encoder. To alleviate the scarcity of temporally annotated data, we present FineLAP-100k, a large-scale synthetic SED dataset constructed through a scalable curation pipeline. Extensive experiments demonstrate that FineLAP achieves SOTA performance across multiple audio understanding tasks, including retrieval, classification, sound event detection, and text-to-audio grounding. Ablation studies further show that coarse- and fine-grained alignment are mutually beneficial, providing insights for building better audio-language models (ALMs).

Paper Structure

This paper contains 52 sections, 5 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Preliminary: Contrastive Language Audio Pretraining
  3. Fine-grained Language-Audio Pretraining
  4. Notation
  5. Model Architecture
  6. Audio Encoder.
  7. Decoupled Audio Adapter.
  8. Text Tower.
  9. Training Objective
  10. Clip-level Sigmoid Loss.
  11. Frame-level Sigmoid Loss.
  12. FineLAP-100k: A Large-scale Synthetic SED Dataset
  13. Overview
  14. Data Creation Pipeline
  15. Audio Event Clipping.
  16. ...and 37 more sections

Figures (6)

  • Figure 1: FineLAP’s performance across different benchmark datasets: Our model achieves state-of-the-art (SOTA) results on both clip- and frame-level tasks.
  • Figure 2: FineLAP effectively leverages the fine-grained alignment between audio frames and textual phrases to detect temporal boundaries of overlapping events.
  • Figure 3: Overview of FineLAP. Left: The decoupled adapter enables simultaneous clip- and frame-level understanding. Right: The framework synergizes global and dense alignment by leveraging heterogeneous supervision via a dual-stream sigmoid loss with cluster-based sampling.
  • Figure 4: Visualization of FineLAP's performance on DESED-Eval
  • Figure 5: Visualization of FineLAP's performance on AudioSet-Strong
  • ...and 1 more figures