Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TACOS: Temporally-aligned Audio CaptiOnS for Language-Audio Pretraining

Paul Primus, Florian Schmid, Gerhard Widmer

TL;DR

The paper addresses the limitation of global, clip-level captions for audio-language models by introducing TACOS, a dataset of temporally aligned captions for audio tracks and a frame-wise contrastive training objective. By learning to align text descriptions with specific temporal regions, the model captures richer temporal structure than traditional CLAP-style approaches. Experimental results show improved temporal text-audio alignment on AudioSet Strong when finetuned with TACOS’s strong captions, and the combination of TACOS weak captions with strong supervision yields additional gains; the work provides both a valuable dataset and a scalable training framework with practical implications for audio retrieval, generation, and QA. The TACOS resources are released publicly, enabling broader adoption and further research into temporally precise audio-language understanding.

Abstract

Learning to associate audio with textual descriptions is valuable for a range of tasks, including pretraining, zero-shot classification, audio retrieval, audio captioning, and text-conditioned audio generation. Existing contrastive language-audio pretrained models are typically trained using global, clip-level descriptions, which provide only weak temporal supervision. We hypothesize that CLAP-like language-audio models - particularly, if they are expected to produce frame-level embeddings - can benefit from a stronger temporal supervision. To confirm our hypothesis, we curate a novel dataset of approximately 12,000 audio recordings from Freesound, each annotated with single-sentence free-text descriptions linked to a specific temporal segment in an audio recording. We use large language models to clean these annotations by removing references to non-audible events, transcribed speech, typos, and annotator language bias. We further propose a frame-wise contrastive training strategy that learns to align text descriptions with temporal regions in an audio recording and demonstrate that our model has better temporal text-audio alignment abilities compared to models trained only on global captions when evaluated on the AudioSet Strong benchmark. The dataset and our source code are available on Zenodo and GitHub, respectively.

TACOS: Temporally-aligned Audio CaptiOnS for Language-Audio Pretraining

TL;DR

The paper addresses the limitation of global, clip-level captions for audio-language models by introducing TACOS, a dataset of temporally aligned captions for audio tracks and a frame-wise contrastive training objective. By learning to align text descriptions with specific temporal regions, the model captures richer temporal structure than traditional CLAP-style approaches. Experimental results show improved temporal text-audio alignment on AudioSet Strong when finetuned with TACOS’s strong captions, and the combination of TACOS weak captions with strong supervision yields additional gains; the work provides both a valuable dataset and a scalable training framework with practical implications for audio retrieval, generation, and QA. The TACOS resources are released publicly, enabling broader adoption and further research into temporally precise audio-language understanding.

Abstract

Learning to associate audio with textual descriptions is valuable for a range of tasks, including pretraining, zero-shot classification, audio retrieval, audio captioning, and text-conditioned audio generation. Existing contrastive language-audio pretrained models are typically trained using global, clip-level descriptions, which provide only weak temporal supervision. We hypothesize that CLAP-like language-audio models - particularly, if they are expected to produce frame-level embeddings - can benefit from a stronger temporal supervision. To confirm our hypothesis, we curate a novel dataset of approximately 12,000 audio recordings from Freesound, each annotated with single-sentence free-text descriptions linked to a specific temporal segment in an audio recording. We use large language models to clean these annotations by removing references to non-audible events, transcribed speech, typos, and annotator language bias. We further propose a frame-wise contrastive training strategy that learns to align text descriptions with temporal regions in an audio recording and demonstrate that our model has better temporal text-audio alignment abilities compared to models trained only on global captions when evaluated on the AudioSet Strong benchmark. The dataset and our source code are available on Zenodo and GitHub, respectively.
Paper Structure (15 sections, 6 equations, 2 figures, 3 tables)

This paper contains 15 sections, 6 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Creation of the Tacos Dataset
  3. Audio Data Collection
  4. Audio Processing
  5. Collecting Temporally-Aligned Captions
  6. Postprocessing Captions
  7. Annotation Statistics and Dataset Splits
  8. Contrastive Training with Temporally-Strong Free-text Annotations
  9. Experiments
  10. Datasets
  11. Audio & Text Embedding
  12. Training
  13. Evaluation Setup
  14. Results and Discussion
  15. Conclusion

Figures (2)

  • Figure 1: Distribution of annotated region durations.
  • Figure 2: PSDS1 scores across a range of temperature values ($\tau$) for models trained with weak and strong annotations in TACOS.