Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Audiocards: Structured Metadata Improves Audio Language Models For Sound Design

Sripathi Sridhar, Prem Seetharaman, Oriol Nieto, Mark Cartwright, Justin Salamon

TL;DR

This work tackles the problem of missing or unstructured metadata in large sound effect libraries by introducing audiocards, a structured, multi-field metadata format grounded in acoustic attributes, sound classes, and usage context. By leveraging world knowledge from LLMs, audiocards enable both improved metadata generation and training of audio-language models, including descriptive captioning and text-audio retrieval, with specialized models like Whisper-Cards and Cards-CLAP outperforming relevant baselines on sound-design tasks. Across professional libraries and general datasets, training on audiocards yields consistent gains in caption quality and retrieval performance, demonstrating the value of domain-aligned metadata for audio understanding. The authors also release ASFx eval and a curated audiocard dataset to catalyze further research in audio-language modeling for sound design and library organization.

Abstract

Sound designers search for sounds in large sound effects libraries using aspects such as sound class or visual context. However, the metadata needed for such search is often missing or incomplete, and requires significant manual effort to add. Existing solutions to automate this task by generating metadata, i.e. captioning, and search using learned embeddings, i.e. text-audio retrieval, are not trained on metadata with the structure and information pertinent to sound design. To this end we propose audiocards, structured metadata grounded in acoustic attributes and sonic descriptors, by exploiting the world knowledge of LLMs. We show that training on audiocards improves downstream text-audio retrieval, descriptive captioning, and metadata generation on professional sound effects libraries. Moreover, audiocards also improve performance on general audio captioning and retrieval over the baseline single-sentence captioning approach. We release a curated dataset of sound effects audiocards to invite further research in audio language modeling for sound design.

Audiocards: Structured Metadata Improves Audio Language Models For Sound Design

TL;DR

This work tackles the problem of missing or unstructured metadata in large sound effect libraries by introducing audiocards, a structured, multi-field metadata format grounded in acoustic attributes, sound classes, and usage context. By leveraging world knowledge from LLMs, audiocards enable both improved metadata generation and training of audio-language models, including descriptive captioning and text-audio retrieval, with specialized models like Whisper-Cards and Cards-CLAP outperforming relevant baselines on sound-design tasks. Across professional libraries and general datasets, training on audiocards yields consistent gains in caption quality and retrieval performance, demonstrating the value of domain-aligned metadata for audio understanding. The authors also release ASFx eval and a curated audiocard dataset to catalyze further research in audio-language modeling for sound design and library organization.

Abstract

Sound designers search for sounds in large sound effects libraries using aspects such as sound class or visual context. However, the metadata needed for such search is often missing or incomplete, and requires significant manual effort to add. Existing solutions to automate this task by generating metadata, i.e. captioning, and search using learned embeddings, i.e. text-audio retrieval, are not trained on metadata with the structure and information pertinent to sound design. To this end we propose audiocards, structured metadata grounded in acoustic attributes and sonic descriptors, by exploiting the world knowledge of LLMs. We show that training on audiocards improves downstream text-audio retrieval, descriptive captioning, and metadata generation on professional sound effects libraries. Moreover, audiocards also improve performance on general audio captioning and retrieval over the baseline single-sentence captioning approach. We release a curated dataset of sound effects audiocards to invite further research in audio language modeling for sound design.
Paper Structure (17 sections, 2 figures, 2 tables)

This paper contains 17 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Audiocards
  3. Fields
  4. Training Data
  5. Human verified sound effects evaluation dataset
  6. Generation
  7. Improving UCS categories estimation with card classification
  8. Models
  9. Audio captioning
  10. Text-audio contrastive representation learning
  11. Experiments
  12. Can audio captioning models trained on audiocards generate structured metadata?
  13. Are captions generated in audiocards better than baseline and LALM captions?
  14. Does generating audiocards lead to better captions?
  15. Does training text-audio models on audiocards improve downstream retrieval performance?
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Left: We propose Audiocards, structured metadata which describes an audio file with attributes relevant to sound designers. We prompt an LLM with the available text metadata and audio descriptors, and generate an audiocard, which can be used for text-based search and to train audio-language models. Right: Audiocard generated by our Whisper-cards audio captioner from input audio without text metadata.
  • Figure 2: Audiocard field metadata prediction evaluated on ASFx eval. We use Audio Flamingo 3 (AF3) goel2025audioflamingo3, a state-of-the-art large audio language model, in think mode to utilize its reasoning capabilities. Whisper-Cards, trained to generate audiocards on our data, achieves superior performance on all audiocard fields.