Music Arena: Live Evaluation for Text-to-Music

TL;DR

This work addresses the lack of scalable, human-ground-truth evaluation and renewable preference data for text-to-music (TTM) by introducing Music Arena, a live evaluation platform where real users perform pairwise comparisons of two TTM outputs. The system uses an LLM-based routing engine to harmonize heterogeneous model interfaces, collects fine-grained listening data and natural language feedback, and enforces a rolling, privacy-preserving data release policy to build a transparent leaderboard. An open-source, Docker-based framework enables integration of both open-weight and API-based TTM systems, and an initial data release demonstrates the platform's viability with a public leaderboard and detailed usage statistics. The work lays a foundation for domain-specific live evaluation, enabling more reliable automatic metrics and repeatable comparisons while upholding ethical and privacy safeguards.

Abstract

We present Music Arena, an open platform for scalable human preference evaluation of text-to-music (TTM) models. Soliciting human preferences via listening studies is the gold standard for evaluation in TTM, but these studies are expensive to conduct and difficult to compare, as study protocols may differ across systems. Moreover, human preferences might help researchers align their TTM systems or improve automatic evaluation metrics, but an open and renewable source of preferences does not currently exist. We aim to fill these gaps by offering *live* evaluation for TTM. In Music Arena, real-world users input text prompts of their choosing and compare outputs from two TTM systems, and their preferences are used to compile a leaderboard. While Music Arena follows recent evaluation trends in other AI domains, we also design it with key features tailored to music: an LLM-based routing system to navigate the heterogeneous type signatures of TTM systems, and the collection of *detailed* preferences including listening data and natural language feedback. We also propose a rolling data release policy with user privacy guarantees, providing a renewable source of preference data and increasing platform transparency. Through its standardized evaluation protocol, transparent data access policies, and music-specific features, Music Arena not only addresses key challenges in the TTM ecosystem but also demonstrates how live evaluation can be thoughtfully adapted to unique characteristics of specific AI domains. Music Arena is available at: https://music-arena.org . Preference data is available at: https://huggingface.co/music-arena .

Figures (5)

• Figure 1: The Music Arena data lifecycle. On the Frontend, users engage in "battles": they submit text prompts, listen to outputs from two music generation systems, and specify their preferences. The central Backend orchestrates the battles: it extracts structured information from text prompts using an LLM to determine model compatibility, routes prompts to appropriate Model Endpoints for generation, delivers music audio to users, and stores the resulting battle data in a Database. Collected data is used to compile a public leaderboard and publicly released on a recurring basis.
• Figure 2: Music Arena leaderboard (July 28 -- Aug 31, 2025), plotting Arena Score (Y-axis) against Generation Speed (Median RTF, X-axis, log scale). Colors and shapes distinguish models by their training data and access type (open weights/proprietary), respectively. This visualization emphasizes the key tradeoff between model quality (score) and interactive latency (speed), an important consideration for creative music applications.
• Figure 3: Distribution of user prompt lengths from voted battles (after removing outliers)
• Figure 4: A word cloud of the most frequent keywords in user prompts from voted battles.
• Figure 5: An example of a completed user battle in the Music Arena frontend.