Designing a Multimodal Viewer for Piano Performance Analysis -- a Pedagogy-First Approach
Joonhyung Bae, Hyeyoon Cho, Kirak Kim, Dawon Park, Taegyun Kwon, Yoon-Seok Choi, Hyeon Hur, Shigeru Kai, Yohei Wada, Satoshi Obata, Akira Maezawa, Jaebum Park, Jonghwa Park, Juhan Nam
TL;DR
This paper tackles the communication barrier in piano pedagogy caused by abstract verbal cues by deriving seven pedagogical data-presentation needs from expert interviews and translating them into a pedagogy-first, multimodal web dashboard. They propose and prototype the Multimodal Piano Dataset Viewer, integrating video, motion capture, and scores with synchronized playback and analysis tools. The study demonstrates technical feasibility using a five-stage workflow and data from a professional and an amateur pianist, and discusses limitations such as limited participant data and lack of educational outcome testing, outlining plans for broader validation. The work advocates a shift from technology-first to education-first design in MIR for music education.
Abstract
Abstract instructions in piano education, such as "raise your wrist" and "relax your tension," lead to varying interpretations among learners, preventing instructors from effectively conveying their intended pedagogical guidance. To address this problem, this study conducted systematic interviews with a piano professor with 18 years teaching experience, and two researchers derived seven core need groups through cross-validation. Based on these findings, we developed a web-based dashboard prototype integrating video, motion capture, and musical scores, enabling instructors to provide concrete, visual feedback instead of relying solely on abstract verbal instructions. Technical feasibility was validated through 109 performance datasets.