From Daily Song to Daily Self: Supporting Reflective Songwriting of Deaf and Hard-of-Hearing Individuals through Generative Music AI

TL;DR

SoulNote, a GenAI system enabling Deaf and Hard-of-Hearing individuals to engage in iterative songwriting, is introduced, demonstrating how GenAI can support marginalized communities by transforming creative expression into a daily practice of self-discovery and reflection.

Abstract

The rapid advancement of generative AI (GenAI) is expanding access to songwriting, offering a new medium of self-expression for Deaf and Hard-of-Hearing (DHH) individuals. However, emerging technologies that support DHH individuals in expressing themselves through music have largely been evaluated in single-session settings and often fall short in helping users unfamiliar with songwriting convey personal narratives or sustain engagement over time. This paper explores songwriting as an extended, music-based journaling practice that supports sustained emotional reflection over multiple sessions. We introduce SoulNote, a GenAI system enabling DHH to engage in iterative songwriting. Grounded in user-centered design, including a design workshop, a preliminary study, and a multi-session diary study, our findings show that ongoing songwriting with \textit{SoulNote} facilitated emotional growth across three dimensions: self-insight, emotion regulation, and \revised{everyday attitudes toward emotions and self-care}. Overall, this work demonstrates how GenAI can support marginalized communities by transforming creative expression into a daily practice of self-discovery and reflection.

Figures (5)

• Figure 1: Overview of the research design and procedures.
• Figure 2: Interfaces and features of SoulNote. (A) Conversation interface for songwriting with four dialogue phases (highlights indicate CA strategies such as supportive and unembellished response (a), imagery-based questioning (b), and context-based music style recommendation (c)), (B) Interactive lyrics editing interface for modifying CA-generated lyrics, (C) Music appreciation interface with visual assistance, and (D) interface for archiving and reviewing previous songwriting results, including conversation summaries and music replay.
• Figure 3: Overall process and theory of change in CA-assisted songwriting. The upper layer (A) represents user-side psychological processes associated with interacting with the CA, organized into three dimensions: developing self-insight, experimenting with emotion regulation strategies, and shifts in everyday attitudes and behaviors. To complement these qualitatively derived mechanisms, we also analyzed behavioral log indicators of system engagement (e.g.,lyric editing time and counts, frequency of initiating difficult topics, contextual music recommendations requested, and replay counts of archived songs; see Section 6). The lower layer (B) illustrates the structure of a typical session with the CA, detailing the agent’s actions (lyric generation, music generation, and archiving) and the CA’s conversational strategies at each step of the process (e.g., narrative and image-based questions). The middle timeline summarizes SoulNote’s overall workflow and the average time users spent in each task phase within a session, including music concept setting, lyric and music creation, appreciation, and discussion.
• Figure 4: Individual songwriting patterns of DHH participants across 12 sessions. The patterns are categorized into five topics, four intents, five lyrics creation strategies, and four musical moods. Topics are annotated with (N) for negative and (P) for positive emotions. Music styles are represented as follows: Ballad (B), Pop (P), Acoustic (A), Folk (F), Classical (CI), CCM (C), Dance (D), Hip-hop (H), Indie (I), Rock (R), and Trot (T).
• Figure 5: Questionnaire results of the comparative experiment across four conditions. Values in parentheses indicate standard deviations. Between-condition differences were assessed using a one-way ANOVA, followed by Tukey’s HSD post-hoc tests for pairwise comparisons (*p < 0.05, **p < 0.01, ***p < 0.001)