Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Coupling the Heart to Musical Machines

Eric Easthope

TL;DR

This work addresses the underexplored role of the heart in musical biofeedback for NIME interfaces. It introduces a 1D, real-time controller that maps PPG-derived heart rate to tempo warping of audio via a Bluetooth HCI, using open-source Python tools. The main contribution is a practical demonstration showing how a heart-rate signal can drive perceptually meaningful tempo changes and form a potential sound-heart-PPG biofeedback loop, with discussions of slow biofeedback ambience and latent-space extensions. Findings indicate that HR-driven tempo modulation yields perceptible tempo changes and supports ambience-oriented performance; the work broadens expressivity and control modalities for live electronic music and motivates integration with non-linear, higher-dimensional mappings.

Abstract

Biofeedback is being used more recently as a general control paradigm for human-computer interfaces (HCIs). While biofeedback especially from breath has seen increasing uptake as a controller for novel musical interfaces, new interfaces for musical expression (NIMEs), the community has not given as much attention to the heart. The heart is just as intimate a part of music as breath and it is argued that the heart determines our perception of time and so indirectly our perception of music. Inspired by this I demonstrate a photoplethysmogram (PPG)-based NIME controller using heart rate as a 1D control parameter to transform the qualities of sounds in real-time over a Bluetooth wireless HCI. I apply time scaling to "warp" audio buffers inbound to the sound card, and play these transformed audio buffers back to the listener wearing the PPG sensor, creating a hypothetical perceptual biofeedback loop: changes in sound change heart rate to change PPG measurements to change sound. I discuss how a sound-heart-PPG biofeedback loop possibly affords greater control and/or variety of movements with a 1D controller, how controlling the space and/or time scale of sound playback with biofeedback makes for possibilities in performance ambience, and I briefly discuss generative latent spaces as a possible way to extend a 1D PPG control space.

Coupling the Heart to Musical Machines

TL;DR

This work addresses the underexplored role of the heart in musical biofeedback for NIME interfaces. It introduces a 1D, real-time controller that maps PPG-derived heart rate to tempo warping of audio via a Bluetooth HCI, using open-source Python tools. The main contribution is a practical demonstration showing how a heart-rate signal can drive perceptually meaningful tempo changes and form a potential sound-heart-PPG biofeedback loop, with discussions of slow biofeedback ambience and latent-space extensions. Findings indicate that HR-driven tempo modulation yields perceptible tempo changes and supports ambience-oriented performance; the work broadens expressivity and control modalities for live electronic music and motivates integration with non-linear, higher-dimensional mappings.

Abstract

Biofeedback is being used more recently as a general control paradigm for human-computer interfaces (HCIs). While biofeedback especially from breath has seen increasing uptake as a controller for novel musical interfaces, new interfaces for musical expression (NIMEs), the community has not given as much attention to the heart. The heart is just as intimate a part of music as breath and it is argued that the heart determines our perception of time and so indirectly our perception of music. Inspired by this I demonstrate a photoplethysmogram (PPG)-based NIME controller using heart rate as a 1D control parameter to transform the qualities of sounds in real-time over a Bluetooth wireless HCI. I apply time scaling to "warp" audio buffers inbound to the sound card, and play these transformed audio buffers back to the listener wearing the PPG sensor, creating a hypothetical perceptual biofeedback loop: changes in sound change heart rate to change PPG measurements to change sound. I discuss how a sound-heart-PPG biofeedback loop possibly affords greater control and/or variety of movements with a 1D controller, how controlling the space and/or time scale of sound playback with biofeedback makes for possibilities in performance ambience, and I briefly discuss generative latent spaces as a possible way to extend a 1D PPG control space.
Paper Structure (11 sections, 1 figure)

This paper contains 11 sections, 1 figure.

Table of Contents

  1. Introduction
  2. Background
  3. Related Work
  4. Objective
  5. Material & Methods
  6. Result
  7. Discussion
  8. Sound-heart-PPG biofeedback loops
  9. "Wandering" controllers with slow biofeedback
  10. Extending "slow" biofeedback control with machine learning, non-linear maps
  11. Conclusion

Figures (1)

  • Figure 1: Audio amplitudes over time for a 5 minute, 38 second song sampled at 44.1 kHz from a WAV file with concurrent (simulated) heart rate data sampled from PPG and a derived "multiplier" for the real-time tempo of the playing audio.