Go witheFlow: Real-time Emotion Driven Audio Effects Modulation
Edmund Dervakos, Spyridon Kantarelis, Vassilis Lyberatos, Jason Liartis, Giorgos Stamou
TL;DR
This work addresses real-time, emotion-driven audio effects modulation by integrating performer biosignals and audio cues into a DAW-friendly system called $witheFlow$. It combines a lightweight local pipeline with a biosignal feature extractor yielding $Attention$ and $Relaxation$, an ECG-based $SI$ stress metric, and an audio emotion regressor operating in the $VA$ (Valence-Arousal) space to drive a rule-based mixer. Key contributions include the modular architecture, a Python/MIDI implementation, configurable YAML mixing rules, and robustness to artifacts, all designed to preserve human creative agency during live performance. The study demonstrates a proof-of-concept with musicians, highlighting enhanced expressive responsiveness and outlining future directions for datasets, explainability, and hybrid edge-cloud deployments to scale emotionally adaptive music technologies.
Abstract
Music performance is a distinctly human activity, intrinsically linked to the performer's ability to convey, evoke, or express emotion. Machines cannot perform music in the human sense; they can produce, reproduce, execute, or synthesize music, but they lack the capacity for affective or emotional experience. As such, music performance is an ideal candidate through which to explore aspects of collaboration between humans and machines. In this paper, we introduce the witheFlow system, designed to enhance real-time music performance by automatically modulating audio effects based on features extracted from both biosignals and the audio itself. The system, currently in a proof-of-concept phase, is designed to be lightweight, able to run locally on a laptop, and is open-source given the availability of a compatible Digital Audio Workstation and sensors.