Jess+: designing embodied AI for interactive music-making
Craig Vear, Johann Benerradi
TL;DR
The paper addresses building embodied AI to enable inclusive live music-making for disabled and non-disabled musicians. It presents Jess+, an intelligent digital score with a robotic arm and an AI Factory that runs multiple neural networks trained on an embodied musicking dataset to drive real-time gestures. Through iterative, user-centered design and four workshops, the study shows that embodied interactions with the AI co-create music and transform participants' practice. The work contributes a modular architecture, a surrogate mind with trains of thought, and a belief system guiding gesture language, offering a path toward accessible, collaborative improvisation and new forms of shared creativity.
Abstract
In this paper, we discuss the conceptualisation and design of embodied AI within an inclusive music-making project. The central case study is Jess+ an intelligent digital score system for shared creativity with a mixed ensemble of non-disabled and disabled musicians. The overarching aim is that the digital score enables disabled musicians to thrive in a live music conversation with other musicians regardless of the potential barriers of disability and music-making. After defining what we mean by embodied AI and how this approach supports the aims of the Jess+ project, we outline the main design features of the system. This includes several novel approaches such as its modular design, an AI Factory based on an embodied musicking dataset, and an embedded belief system. Our findings showed that the implemented design decisions and embodied-AI approach led to rich experiences for the musicians which in turn transformed their practice as an inclusive ensemble.