Designing a Generative AI-Assisted Music Psychotherapy Tool for Deaf and Hard-of-Hearing Individuals

TL;DR

A music psychotherapy tool co-designed with therapists, integrating conversational agents (CAs) and music generative AI as symbolic and therapeutic media is presented, finding that collaborative song writing with the CA enabled them to experience emotional release, reinterpretation, and deeper self-understanding.

Abstract

Songwriting has long served as a powerful medium for expressing unconscious emotions and fostering self-awareness in psychotherapy. Due to the auditory-centric nature of traditional approaches, Deaf and Hard-of-Hearing (DHH) individuals have often been excluded from music's therapeutic benefits. In response, this study presents a music psychotherapy tool co-designed with therapists, integrating conversational agents (CAs) and music generative AI as symbolic and therapeutic media. Through a usage study with 23 DHH individuals, we found that collaborative song writing with the CA enabled them to experience emotional release, reinterpretation, and deeper self-understanding. In particular, the CA's strategies -- supportive empathy, example response options, and visual-based metaphors -- were found to facilitate musical dialogue effectively for DHH individuals. These findings contribute to inclusive AI design by showing the potential of human-AI collaboration to bridge therapeutic artistic practices.

Figures (7)

• Figure 1: Design workshop process. Sessions comprise persona design, a discussion about GenAI-based technology, and process design. In the design session, the final design emerged from group discussion, led by the main facilitator (T1, red), after individual design work.
• Figure 2: States and representative Q&A examples of the music psychotherapy assistive tool. The process consists of four states: Therapeutic connection (A), Making lyrics (B), Making music (C), and Song discussion (D). Each state contains the steps and required variables that serve as the criteria for proceeding to the next state. Examples of the CA's three conversational strategies are indicated using color coding - supportive empathy (pink), example response option (green), and visual-based metaphor and analogy (yellow).
• Figure 3: The interface of a music psychotherapy assistive tool. The left panel shows the CA-based conversational interface for songwriting, structured into four states (therapeutic connection, making lyrics, making music, and song discussion), while the right panel displays the generated music rendered as visualizations for appreciation, along with the corresponding music–visual mapping elements.
• Figure 4: Overall process of the prompt generator. The CA tool operates on a state-step framework involving general, state guidance, and a required variable extractor prompt. The specific prompting for the prompt generator can be found in Appendix \ref{['adx: prompt engineering']}.
• Figure 5: Individual time spent by participants in each songwriting state. Participants spent an average of 5.07 minutes (SD=3.00) building rapport with the CA, followed by 13.13 minutes (SD=8.59) discussing lyrics and 14.40 minutes (SD=7.17) discussing musical elements such as genre. They then spend 7.55 minutes (SD=5.56) sharing thoughts and emotions about the generated music.