musicolors: Bridging Sound and Visuals For Synesthetic Creative Musical Experience
ChungHa Lee, Jin-Hyuk Hong
TL;DR
This work addresses how to bridge sound and visuals to support synesthetic creative experiences during music listening. It introduces musicolors, a real-time web-based library that maps pitch, energy, and timbre to colored, sized, and textured visuals, and validates its utility through a qualitative study with eight participants across composers, developers, and listeners. Key contributions include concrete mapping rules, design guidelines for future music visualizations, and insights on integrating visualization with music platforms to enhance inspiration, perceptual richness, and shared cultural practices. The approach holds practical significance for expanding how people experience, create, and communicate music through multimodal visualization.
Abstract
Music visualization is an important medium that enables synesthetic experiences and creative inspiration. However, previous research focused mainly on the technical and theoretical aspects, overlooking users' everyday interaction with music visualizations. This gap highlights the pressing need for research on how music visualization influences users in synesthetic creative experiences and where they are heading. Thus, we developed musicolors, a web-based music visualization library available in real-time. Additionally, we conducted a qualitative user study with composers, developers, and listeners to explore how they use musicolors to appreciate and get inspiration and craft the music-visual interaction. The results show that musicolors provides a rich value of music visualization to users through sketching for musical ideas, integrating visualizations with other systems or platforms, and synesthetic listening. Based on these findings, we also provide guidelines for future music visualizations to offer a more interactive and creative experience.