Robotic Blended Sonification: Consequential Robot Sound as Creative Material for Human-Robot Interaction

TL;DR

The paper addresses how to move beyond masking robot sounds or using abstract sonifications by treating consequential robot sound as a usable material for interaction. It proposes Robotic Blended Sonification, a real-time pipeline that captures robot-generated sounds with electromagnetic field microphones, processes and blends them in a DAW, and replays them as collaborators interact with the robot via a Grasshopper/Rhino setup and OSC controlling the robot's Tool Center Point. The contributions include a concrete method for real-time sound capture and processing of consequential sounds and a framework to explore these sounds through direct human-robot interaction, enabling artistic prototyping and tacit knowledge elicitation. This approach has potential to enrich non-visual feedback modalities in HRI and expand the sonic design space for industrial and service robots.

Abstract

Current research in robotic sounds generally focuses on either masking the consequential sound produced by the robot or on sonifying data about the robot to create a synthetic robot sound. We propose to capture, modify, and utilise rather than mask the sounds that robots are already producing. In short, this approach relies on capturing a robot's sounds, processing them according to contextual information (e.g., collaborators' proximity or particular work sequences), and playing back the modified sound. Previous research indicates the usefulness of non-semantic, and even mechanical, sounds as a communication tool for conveying robotic affect and function. Adding to this, this paper presents a novel approach which makes two key contributions: (1) a technique for real-time capture and processing of consequential robot sounds, and (2) an approach to explore these sounds through direct human-robot interaction. Drawing on methodologies from design, human-robot interaction, and creative practice, the resulting 'Robotic Blended Sonification' is a concept which transforms the consequential robot sounds into a creative material that can be explored artistically and within application-based studies.

Figures (1)

• Figure 1: The approach to Robotic Blended Sonification relies on (1) using electromagnetic field microphones that capture audio vibrations through solid objects, (2) processing the audio, and (3) reproducing the audio in real-time as the collaborator interacts with the robot by guiding the position of the TCP.