PAGURI: a user experience study of creative interaction with text-to-music models

TL;DR

This study investigates how musicians and practitioners interact with text-to-music models through the PAGURI interface, combining prompt-based generation with audio-based personalization via DreamBooth using AudioLDM2. It uses a mixed-method design with 24 participants to assess usability, creativity, and integration into practice, revealing that personalization enhances perceived sound signature and rhythmic expressivity but raises concerns about prompt ambiguity, controllability, and copyright. The findings underscore the need for multimodal control, more diverse training data, and subjective usability metrics beyond traditional audio quality, to better embed TTMs in real-world workflows. Overall, TTMs show promise as inspirational and educational tools, provided future work improves user control, workflow compatibility, and ethical guidelines for personalization.

Abstract

In recent years, text-to-music models have been the biggest breakthrough in automatic music generation. While they are unquestionably a showcase of technological progress, it is not clear yet how they can be realistically integrated into the artistic practice of musicians and music practitioners. This paper aims to address this question via Prompt Audio Generation User Research Investigation (PAGURI), a user experience study where we leverage recent text-to-music developments to study how musicians and practitioners interact with these systems, evaluating their satisfaction levels. We developed an online tool through which users can generate music samples and/or apply recently proposed personalization techniques based on fine-tuning to allow the text-to-music model to generate sounds closer to their needs and preferences. Using semi-structured interviews, we analyzed different aspects related to how participants interacted with the proposed tool to understand the current effectiveness and limitations of text-to-music models in enhancing users' creativity. Our research centers on user experiences to uncover insights that can guide the future development of TTM models and their role in AI-driven music creation. Additionally, they offered insightful perspectives on potential system improvements and their integration into their music practices. The results obtained through the study reveal the pros and cons of the use of TTMs for creative endeavors. Participants recognized the system's creative potential and appreciated the usefulness of its personalization features. However, they also identified several challenges that must be addressed before TTMs are ready for real-world music creation, particularly issues of prompt ambiguity, limited controllability, and integration into existing workflows.

Figures (6)

• Figure 1: The user interface employed to carry out the PAGURI study. The upper part corresponds to prompt-based music generation using AudioLDM2, while the bottom part corresponds to the personalization of the model via DreamBooth, with the possibility to upload audio samples of choice.
• Figure 2: Block diagram of the experimental procedure in the PAGURI user study.
• Figure 3: Diverging bar chart showing the musical knowledge and experiences with AI tools survey answers.
• Figure 4: Diverging bar chart showing the Model Evaluation Survey answers. N.B. In this case, the value on the x-axis corresponds to the number of responses.
• Figure 5: Answers of P5 (a,c,e) and P16 (b,d,f) to the questions Q21, Q22, and Q23 of the Model Evaluation Survey, for each single interaction with the model. The red color indicates that at the specific iteration, the user personalized the model, the blue one, that personalization was not used.