Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking Music Captioning with Music Metadata LLMs

Irmak Bukey, Zhepei Wang, Chris Donahue, Nicholas J. Bryan

TL;DR

This work tackles the problem of limited high-quality music captions by proposing a metadata-driven captioning pipeline that first predicts structured metadata from audio and then converts that metadata into expressive captions using a pre-trained LLM at inference. The approach achieves comparable performance to end-to-end captioners while reducing training time, and offers flexible post-hoc stylization and robust metadata imputation through partial metadata. Key contributions include (1) a two-stage audio-to-metadata-to-caption framework, (2) demonstration of substantial stylization flexibility without retraining, and (3) effective metadata imputation capabilities enabled by partial metadata inputs. This metadata-centric method holds practical impact for music organization, controllable captioning, and scalable data augmentation for downstream music understanding tasks.

Abstract

Music captioning, or the task of generating a natural language description of music, is useful for both music understanding and controllable music generation. Training captioning models, however, typically requires high-quality music caption data which is scarce compared to metadata (e.g., genre, mood, etc.). As a result, it is common to use large language models (LLMs) to synthesize captions from metadata to generate training data for captioning models, though this process imposes a fixed stylization and entangles factual information with natural language style. As a more direct approach, we propose metadata-based captioning. We train a metadata prediction model to infer detailed music metadata from audio and then convert it into expressive captions via pre-trained LLMs at inference time. Compared to a strong end-to-end baseline trained on LLM-generated captions derived from metadata, our method: (1) achieves comparable performance in less training time over end-to-end captioners, (2) offers flexibility to easily change stylization post-training, enabling output captions to be tailored to specific stylistic and quality requirements, and (3) can be prompted with audio and partial metadata to enable powerful metadata imputation or in-filling--a common task for organizing music data.

Rethinking Music Captioning with Music Metadata LLMs

TL;DR

This work tackles the problem of limited high-quality music captions by proposing a metadata-driven captioning pipeline that first predicts structured metadata from audio and then converts that metadata into expressive captions using a pre-trained LLM at inference. The approach achieves comparable performance to end-to-end captioners while reducing training time, and offers flexible post-hoc stylization and robust metadata imputation through partial metadata. Key contributions include (1) a two-stage audio-to-metadata-to-caption framework, (2) demonstration of substantial stylization flexibility without retraining, and (3) effective metadata imputation capabilities enabled by partial metadata inputs. This metadata-centric method holds practical impact for music organization, controllable captioning, and scalable data augmentation for downstream music understanding tasks.

Abstract

Music captioning, or the task of generating a natural language description of music, is useful for both music understanding and controllable music generation. Training captioning models, however, typically requires high-quality music caption data which is scarce compared to metadata (e.g., genre, mood, etc.). As a result, it is common to use large language models (LLMs) to synthesize captions from metadata to generate training data for captioning models, though this process imposes a fixed stylization and entangles factual information with natural language style. As a more direct approach, we propose metadata-based captioning. We train a metadata prediction model to infer detailed music metadata from audio and then convert it into expressive captions via pre-trained LLMs at inference time. Compared to a strong end-to-end baseline trained on LLM-generated captions derived from metadata, our method: (1) achieves comparable performance in less training time over end-to-end captioners, (2) offers flexibility to easily change stylization post-training, enabling output captions to be tailored to specific stylistic and quality requirements, and (3) can be prompted with audio and partial metadata to enable powerful metadata imputation or in-filling--a common task for organizing music data.
Paper Structure (12 sections, 1 figure, 4 tables)

This paper contains 12 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Relevant Work
  3. Method
  4. Metadata prediction model
  5. Metadata-to-caption conversion
  6. Metadata imputation
  7. Experiments
  8. Experimental setup
  9. Evaluation tasks
  10. Style and prompting variations
  11. Results and Analysis
  12. Conclusion

Figures (1)

  • Figure 1: (Top left) Dataset format. Our training dataset consists of paired music and metadata annotations. (Bottom left) Metadata-to-caption synthesis. A text-based LLM is used to convert music metadata into natural language descriptions. (Top right) Inference pipeline for typical music captioning methods where a caption MLLM is trained to generate captions. Optionally, a text-based LLM can be used to extract medatata from generated captions. (Bottom right) Inference pipeline for the proposed metadata MLLM that predicts all or missing metadata fields. Optionally, a text-based LLM can be used to convert the predicted metadata into captions.