ConceptCaps -- a Distilled Concept Dataset for Interpretability in Music Models

TL;DR

ConceptCaps addresses a key barrier in concept-based interpretability for music models by creating a large, coherent dataset of 23k music-description pairs with explicit concept labels. The authors decouple semantic modeling from text generation via a two-stage pipeline: a β-VAE models plausible attribute co-occurrences, and a fine-tuned LLM converts attributes into professional captions, followed by MusicGen audio synthesis conditioned on these captions. They validate quality through linguistic metrics (BLEU, ROUGE, BERTScore, MAUVE), audio-text alignment (CLAP), and TCAV analyses showing musically meaningful concept separability, while achieving efficiency gains over API-based baselines. This pipeline provides fine-grained controllability over concept composition and yields a reproducible, copyright-free resource to advance interpretability research in music AI. The work highlights practical implications for robust, interpretable music-model evaluations and points to future expansion into non-Western musical concepts and broader notion coverage.

Abstract

Concept-based interpretability methods like TCAV require clean, well-separated positive and negative examples for each concept. Existing music datasets lack this structure: tags are sparse, noisy, or ill-defined. We introduce ConceptCaps, a dataset of 23k music-caption-audio triplets with explicit labels from a 200-attribute taxonomy. Our pipeline separates semantic modeling from text generation: a VAE learns plausible attribute co-occurrence patterns, a fine-tuned LLM converts attribute lists into professional descriptions, and MusicGen synthesizes corresponding audio. This separation improves coherence and controllability over end-to-end approaches. We validate the dataset through audio-text alignment (CLAP), linguistic quality metrics (BERTScore, MAUVE), and TCAV analysis confirming that concept probes recover musically meaningful patterns. Dataset and code are available online.

Figures (6)

• Figure 1: Overview of the three-stage dataset generation pipeline. Stage 1 (Semantic Modeling via VAE): A Variational Autoencoder samples from the latent space $z \sim \mathcal{N}(0, I)$ and decodes to generate coherent attribute lists (e.g., "folk, acoustic guitar, upbeat"). Stage 2 (Text Generation via Fine-Tuned LLM): The fine-tuned language model (Llama 3.1 8B) receives the attribute list and generates a professional, descriptive music caption capturing the semantic content. Stage 3 (Audio Synthesis): MusicGen synthesizes copyright-free audio from the generated description, completing the dataset sample with well-aligned audio-text pairs and explicit concept labels suitable for interpretability analysis.
• Figure 2: Visualisation of VAE latent space. Encoder successfully learns to map music genres to latent space. We can also notice music genres "blending" together, especially in "rock" and "pop" genres, where conceptual overlap in source dataset is substantial.
• Figure 3: Individual tag frequency distributions comparing MusicCaps (blue) and our distilled dataset (orange). The distilled dataset preserves source dataset characteristics, resulting in real-world data distribution.
• Figure 4: Comparison of per-sample aspect count distribution in MusicCaps (blue) versus our distilled dataset (orange). Lack of long tail shows improvement over sparse or redundant tags in original dataset.
• Figure 5: CLAP score distributions comparing our distilled dataset against MusicCaps and LP-MusicCaps (left), alongside an ablation study of LLM configurations (right). Higher scores denote superior semantic alignment between audio and text.