SongSage: A Large Musical Language Model with Lyric Generative Pre-training

TL;DR

This work surveys the limits of general-purpose LLMs in lyric-centric and playlist understanding and introduces SongSage, a lyric-focused large musical language model pretrained on LyricBank (5.48B tokens) and fine-tuned with LyricBank-SFT (775k samples across nine tasks). It also presents PlaylistSense, a 5k playlist benchmark with ten real-world perspectives to evaluate playlist and lyric comprehension, revealing substantial gaps and biases in existing models. SongSage demonstrates strong lyric understanding, effective query rewriting for playlist recommendations, superior lyric generation capabilities, and competitive performance on general knowledge benchmarks, while remaining open-source with accompanying training scripts. The work highlights the potential for domain-specific pretraining in music AI and sets the stage for future multimodal extensions and bias-focused analyses to broaden applicability and fairness in music-centric AI systems.

Abstract

Large language models have achieved significant success in various domains, yet their understanding of lyric-centric knowledge has not been fully explored. In this work, we first introduce PlaylistSense, a dataset to evaluate the playlist understanding capability of language models. PlaylistSense encompasses ten types of user queries derived from common real-world perspectives, challenging LLMs to accurately grasp playlist features and address diverse user intents. Comprehensive evaluations indicate that current general-purpose LLMs still have potential for improvement in playlist understanding. Inspired by this, we introduce SongSage, a large musical language model equipped with diverse lyric-centric intelligence through lyric generative pretraining. SongSage undergoes continual pretraining on LyricBank, a carefully curated corpus of 5.48 billion tokens focused on lyrical content, followed by fine-tuning with LyricBank-SFT, a meticulously crafted instruction set comprising 775k samples across nine core lyric-centric tasks. Experimental results demonstrate that SongSage exhibits a strong understanding of lyric-centric knowledge, excels in rewriting user queries for zero-shot playlist recommendations, generates and continues lyrics effectively, and performs proficiently across seven additional capabilities. Beyond its lyric-centric expertise, SongSage also retains general knowledge comprehension and achieves a competitive MMLU score. We will keep the datasets inaccessible due to copyright restrictions and release the SongSage and training script to ensure reproducibility and support music AI research and applications, the datasets release plan details are provided in the appendix.

Figures (13)

• Figure 1: Overview of the PlaylistSense evaluation. In the data construction process, Deepseek generates 10 queries, covering both Music-Based and User-Based perspectives. During evaluation, the evaluator generate descriptions for each playlist, forming a pool. Each query retrieves descriptions from the pool via an embedding model, with success determined by whether the retrieved description matches the query's source playlist.
• Figure 2: Overview of SongSage's three-stage training, detailing the progression from general lyrical knowledge on LyricBank to task-specific instruction tuning with LyricBank-SFT, followed by preference optimization, alongside a breakdown of knowledge domains and task distributions. The logo of SongSage is created using GPT-4.
• Figure 3: Performance of LLMs in generating playlist descriptions, with GPT-4o as the judge model. The numbers represent the number of times the model was selected.
• Figure 4: Accuracy(%) on MusicTheoryBench. The horizontal dashed line represents the random guess score.
• Figure 5: POS frequency distribution in Qwen2.5-Base and SongSage-Base. The numbers represent the average occurrences of each POS per sentence.