herakoi: a sonification experiment for astronomical data
Michele Ginolfi, Luca Di Mascolo, Anita Zanella
TL;DR
Data in astronomy is often visually dominant and limited to the visible spectrum, motivating the need for alternative interpretation methods. The paper presents herakoi, an open-source, real-time hand-tracking sonification tool that maps image regions touched by the hand to MIDI audio, enabling auditory access to visual data. It demonstrates educational and outreach value through demonstrations and a school study, with color recognition accuracies up to 93% and positive engagement among BVI users. It envisions future integration with large language and vision models to provide verbal explanations and interactive dialogue, broadening accessibility and interactive exploration of astronomical data.
Abstract
Recent research is revealing data-sonification as a promising complementary approach to vision, benefiting both data perception and interpretation. We present herakoi, a novel open-source software that uses machine learning to allow real-time image sonification, with a focus on astronomical data. By tracking hand movements via a webcam and mapping them to image coordinates, herakoi translates visual properties into sound, enabling users to "hear" images. Its swift responsiveness allows users to access information in astronomical images with short training, demonstrating high reliability and effectiveness. The software has shown promise in educational and outreach settings, making complex astronomical concepts more engaging and accessible to diverse audiences, including blind and visually impaired individuals. We also discuss future developments, such as the integration of large language and vision models to create a more interactive experience in interpreting astronomical data.