Interactive Sonification for Health and Energy using ChucK and Unity

TL;DR

The paper addresses the need for user-controlled interactive sonification of health and energy data by introducing an end-to-end framework built on ChucK, Unity, and Chunity. It extends prior toolkits with features such as discrete events, interleaved playback, and cross-attribute FM mappings, enabling real-time experimentation and richer audio-visual representations. Through case studies on EEG alpha data and air-quality pollutants from ICAD, the authors demonstrate replication of prior sonic designs alongside new interactive capabilities and synchronized 2D/3D visuals. The work offers a portable, reusable approach that lowers the barrier for domain experts to deploy interactive sonifications, with potential for real-time data integration and broader evaluation.

Abstract

Sonification can provide valuable insights about data but most existing approaches are not designed to be controlled by the user in an interactive fashion. Interactions enable the designer of the sonification to more rapidly experiment with sound design and allow the sonification to be modified in real-time by interacting with various control parameters. In this paper, we describe two case studies of interactive sonification that utilize publicly available datasets that have been described recently in the International Conference on Auditory Display (ICAD). They are from the health and energy domains: electroencephalogram (EEG) alpha wave data and air pollutant data consisting of nitrogen dioxide, sulfur dioxide, carbon monoxide, and ozone. We show how these sonfications can be recreated to support interaction utilizing a general interactive sonification framework built using ChucK, Unity, and Chunity. In addition to supporting typical sonification methods that are common in existing sonification toolkits, our framework introduces novel methods such as supporting discrete events, interleaved playback of multiple data streams for comparison, and using frequency modulation (FM) synthesis in terms of one data attribute modulating another. We also describe how these new functionalities can be used to improve the sonification experience of the two datasets we have investigated.

Figures (6)

• Figure 1: The main track layout of the sonification framework when a dataset consisting of 4 data attributes is first loaded with the default configuration.
• Figure 2: Frequency modulation by EEG data with a minimum frequency of 261.6 Hz and a frequency range of 600 Hz.
• Figure 3: The configuration of discrete sonification of the EEG data with envelope parameters modifying the shape of the sound and threshold parameters to specify the conditions for discrete sounds to be played.
• Figure 4: The configuration to replicate the sound design of the sonification of air quality data, where the first row of each data attribute serves as the modulator, and the second is the carrier for FM synthesis.
• Figure 5: In the audio visualization view, the panning of the sonification of air quality data could be replicated and configured. The center is the location of the listener. The differently coloured speakers refer to the data attributes with the corresponding colours, and their locations indicate the panning; For example, the green speaker refers to O3 in Figure \ref{['fig:air_replicate']} and its panning location is on the far right.