Make the Unhearable Visible: Exploring Visualization for Musical Instrument Practice

TL;DR

This paper tackles the challenge of providing practice-oriented feedback to musicians by making unhearable patterns visible through a design-space exploration of 33 MIDI/audio visualizations. It blends autobiographical expertise with UX research over five years, and employs qualitative case studies and expert interviews to demonstrate that visualization can reveal subtle patterns useful for learning and teaching. Key contributions include: a structured design space for instrument-practice visualization, concrete case-study designs (across note duration, timing consistency, accents, improvisation, and fretboard movement), and design considerations for implementing such tools in real-world education. The work also discusses practical trade-offs, listener- and memory-related limitations, and how visualizations can complement but not replace auditory feedback, with published code and a web app to support reproducibility and further exploration. Overall, the findings suggest substantial potential for real-time and post-hoc visualization to support musical practice and education, while outlining a path toward broader, field-based validation.

Abstract

We explore the potential of visualization to support musicians in instrument practice through real-time feedback and reflection on their playing. Musicians often struggle to observe the patterns in their playing and interpret them with respect to their goals. Our premise is that these patterns can be made visible with interactive visualization: we can make the unhearable visible. However, understanding the design of such visualizations is challenging: the diversity of needs, including different instruments, skills, musical attributes, and genres, means that any single use case is unlikely to illustrate the broad potential and opportunities. To address this challenge, we conducted a design exploration study where we created and iterated on 33 designs, each focusing on a subset of needs, for example, only one musical skill. Our designs are grounded in our own experience as musicians and the ideas and feedback of 18 musicians with various musical backgrounds and we evaluated them with 13 music learners and teachers. This paper presents the results of our exploration, focusing on a few example designs as instances of possible instrument practice visualizations. From our work, we draw design considerations that contribute to future research and products for visual instrument education.

Figures (24)

• Figure 1: An overview of the 33 designs we created to consider different data attributes, musical instrument skills, and visual encodings. They are categorized by the primary skill they support (see \ref{['sec:isolationofskills']}). Some of these designs are described in \ref{['sec:design-exploration']}, all are provided in the supplement.
• Figure 2: Note duration pie charts for perfectly played quarter, half, dotted half, and whole note.
• Figure 3: Note duration bar chart showing seven attempts at playing dotted quarter and two at a whole note.
• Figure 4: Aligned stacking of repetitions.
• Figure 5: Tick encoding with light gray tolerance areas around the target times.