Loop Copilot: Conducting AI Ensembles for Music Generation and Iterative Editing

TL;DR

The paper addresses the challenge of orchestrating multiple AI subsystems for iterative music creation by introducing Loop Copilot, an LLM-driven controller that selects and chains specialized backends while maintaining musical coherence via a Global Attribute Table (GAT). It formalizes the interaction with a stateful framework, demonstrates a two-stage generation/editing workflow, and presents a training-free method for iterative editing through model chaining. An empirical evaluation with eight participants using SUS and TAM reveals generally favorable usability and acceptance, while qualitative feedback identifies limitations in control granularity and integration with existing workflows. The work highlights the potential of conversational, multi-model orchestration to democratize music creation and suggests future work on richer editing tasks, DAW integration, and voice-based interactions to broaden impact.

Abstract

Creating music is iterative, requiring varied methods at each stage. However, existing AI music systems fall short in orchestrating multiple subsystems for diverse needs. To address this gap, we introduce Loop Copilot, a novel system that enables users to generate and iteratively refine music through an interactive, multi-round dialogue interface. The system uses a large language model to interpret user intentions and select appropriate AI models for task execution. Each backend model is specialized for a specific task, and their outputs are aggregated to meet the user's requirements. To ensure musical coherence, essential attributes are maintained in a centralized table. We evaluate the effectiveness of the proposed system through semi-structured interviews and questionnaires, highlighting its utility not only in facilitating music creation but also its potential for broader applications.

Figures (4)

• Figure 1: A conceptual illustration of interaction with Loop Copilot. The diagram depicts a two-round conversation: initially, a user requests music generation and the AI provides a loop. In the subsequent round, the user seeks modifications, and the AI offers a refined loop, emphasizing Loop Copilot's iterative feedback-driven music creation process.
• Figure 2: The diagram of Loop Copilot's workflow. Once the user inputs the request, firstly, Loop Copilot preprocesses the input and converts it to textual modality; secondly, the LLM, based on the input, the system principles, and the chat history, performs the task analysis and calls the corresponding models; after that, the backend models execute the task and output the result; finally, the LLM does the final processing of the output and returns it.
• Figure 3: The box plot depicting SUS score results with an average of 75.31$\pm$15.32. The dotted line marks the threshold for effectiveness.
• Figure 4: Box plot of the TAM score results. Perceived Usefulness (PU) with an average of 3.58$\pm$1.13; Perceived Ease of Use (PEOU) averaging 3.89$\pm$0.80, Overall TAM score of 4.09$\pm$1.09. These scores reflect participants' favorable perceptions of the system's utility and usability.