Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Erie: A Declarative Grammar for Data Sonification

Hyeok Kim, Yea-Seul Kim, Jessica Hullman

TL;DR

Erie addresses the lack of expressive software for data sonification by introducing a declarative grammar that decouples auditory design from visuals. It defines a data-to-sound pipeline with top-level specs, streams, tones, encodings, and multi-stream composition, and provides web-based tooling (spec API, queue compiler, and Web Audio/Speech player) to render designs. The approach is demonstrated through replication of prior use cases and an interactive design gallery, highlighting Erie’s independence from visual encodings, expressiveness, data-driven syntax, extensibility, and standards compatibility. The work sets a foundation for research into perceptual effectiveness, interactive sonification, and cross-environment tooling, with future plans for streaming data, interactivity, and intelligent design recommendations.

Abstract

Data sonification-mapping data variables to auditory variables, such as pitch or volume-is used for data accessibility, scientific exploration, and data-driven art (e.g., museum exhibitions) among others. While a substantial amount of research has been made on effective and intuitive sonification design, software support is not commensurate, limiting researchers from fully exploring its capabilities. We contribute Erie, a declarative grammar for data sonification, that enables abstractly expressing auditory mappings. Erie supports specifying extensible tone designs (e.g., periodic wave, sampling, frequency/amplitude modulation synthesizers), various encoding channels, auditory legends, and composition options like sequencing and overlaying. Using standard Web Audio and Web Speech APIs, we provide an Erie compiler for web environments. We demonstrate the expressiveness and feasibility of Erie by replicating research prototypes presented by prior work and provide a sonification design gallery. We discuss future steps to extend Erie toward other audio computing environments and support interactive data sonification.

Erie: A Declarative Grammar for Data Sonification

TL;DR

Erie addresses the lack of expressive software for data sonification by introducing a declarative grammar that decouples auditory design from visuals. It defines a data-to-sound pipeline with top-level specs, streams, tones, encodings, and multi-stream composition, and provides web-based tooling (spec API, queue compiler, and Web Audio/Speech player) to render designs. The approach is demonstrated through replication of prior use cases and an interactive design gallery, highlighting Erie’s independence from visual encodings, expressiveness, data-driven syntax, extensibility, and standards compatibility. The work sets a foundation for research into perceptual effectiveness, interactive sonification, and cross-environment tooling, with future plans for streaming data, interactivity, and intelligent design recommendations.

Abstract

Data sonification-mapping data variables to auditory variables, such as pitch or volume-is used for data accessibility, scientific exploration, and data-driven art (e.g., museum exhibitions) among others. While a substantial amount of research has been made on effective and intuitive sonification design, software support is not commensurate, limiting researchers from fully exploring its capabilities. We contribute Erie, a declarative grammar for data sonification, that enables abstractly expressing auditory mappings. Erie supports specifying extensible tone designs (e.g., periodic wave, sampling, frequency/amplitude modulation synthesizers), various encoding channels, auditory legends, and composition options like sequencing and overlaying. Using standard Web Audio and Web Speech APIs, we provide an Erie compiler for web environments. We demonstrate the expressiveness and feasibility of Erie by replicating research prototypes presented by prior work and provide a sonification design gallery. We discuss future steps to extend Erie toward other audio computing environments and support interactive data sonification.
Paper Structure (46 sections, 28 equations, 4 figures, 10 tables)

This paper contains 46 sections, 28 equations, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Data Sonification
  4. Sonification Tools and Toolkits
  5. Declarative Grammar
  6. Gaps in Sonification Development Practices
  7. Design Considerations
  8. Erie Grammar
  9. A Walkthrough Example
  10. Top-Level Specification and Stream
  11. Data, dataset, and Transform
  12. Tone
  13. Encoding
  14. Scale
  15. Composition
  16. ...and 31 more sections

Figures (4)

  • Figure 1: The formal definition of Erie. For applicable elements, roughly analogous visualization elements are denoted by $\approx$ signs.
  • Figure 2: Our replication and extension of Audio Narrative audioNarrative:siu2022 using Erie. In addition to the originally offered sequencing and speech description, we included options for using different encoding channels (A) and playing the scale description (B).
  • Figure 3: Our replication and extension of Chart Reader thompson2023:chartreader using Erie. We further included user options for toggling the hover/selection interaction (A) and aggregation (B).
  • Figure 4: Our replication of the Nine Rounds a Second article vegas using Erie.