Hearing the Robot's Mind: Sonification for Explicit Feedback in Human-Robot Interaction

TL;DR

Results indicate that while sonification improves the robot's expressivity and communication effectiveness, the design of the auditory feedback needs refinement to enhance user experience, underscore the importance of carefully designed auditory feedback in developing more effective and engaging HRI systems.

Abstract

Social robots are required not only to understand human intentions but also to effectively communicate their intentions or own internal states to users. This study explores the use of sonification to provide explicit auditory feedback, enhancing mutual understanding in HRI. We introduce a novel sonification approach that conveys the robot's internal state, linked to its perception of nearby individuals and their interaction intentions. The approach is evaluated through a two-fold user study: an online video-based survey with $26$ participants and live experiments with $10$ participants. Results indicate that while sonification improves the robot's expressivity and communication effectiveness, the design of the auditory feedback needs refinement to enhance user experience. Participants found the auditory cues useful but described the sounds as uninteresting and unpleasant. These findings underscore the importance of carefully designed auditory feedback in developing more effective and engaging HRI systems.

Figures (7)

• Figure 1: A robot offering chocolate treats to a visitor of a public building uses audio cues to inform nearby people of its internal state.
• Figure 2: Experimental setup with a zoomed view of the sensor and the speaker used to elaborate our two-way communication between a robot and a user.
• Figure 3: Plots of the piecewise linear transfer functions used to calculate the sound parameters: volume and pitch depend on $p$, whereas the amount of vibrato effect depends on its rate of change.
• Figure 4: Diagram of the sound synthesis pipeline
• Figure 5: Spectrogram of the sound feedback during an interaction sequence in which one user approaches the robot looking very interested (a), the robot keeps offering them the chocolate for a long time (b, from second 4 to second 10), then the user suddenly leaves without actually taking the chocolate (c, second 10 to 12). This sequence corresponds to timestamps 0:39 to 0:55 of the attached video https://youtu.be/Cn9dQBznWzY