Music Mode: Transforming Robot Movement into Music Increases Likability and Perceived Intelligence

TL;DR

Music Mode maps a robot's joint velocities to audio samples to generate music as the robot moves. Two controlled experiments and an embedded case study show that a synchronized Orchestra soundscape increases perceived safety, animacy, anthropomorphism, likeability, and perceived intelligence compared with native or random sounds, with movement–sound coupling driving the strongest effects. A follow-up online test rules out music alone as the sole driver, underscoring the importance of movement-linked audio for perceived intelligence. The work documents an interdisciplinary design process, a scalable on-board implementation, and open-source resources to guide expressive robotic sound in everyday, human-facing environments.

Abstract

As robots enter everyday spaces like offices, the sounds they create affect how they are perceived. We present Music Mode, a novel mapping between a robot's joint motions and sounds, programmed by artists and engineers to make the robot generate music as it moves. Two experiments were designed to characterize the effect of this musical augmentation on human users. In the first experiment, a robot performed three tasks while playing three different sound mappings. Results showed that participants observing the robot perceived it as more safe, animate, intelligent, anthropomorphic, and likable when playing the Music Mode Orchestra software. To test whether the results of the first experiment were due to the Music Mode algorithm, rather than music alone, we conducted a second experiment. Here the robot performed the same three tasks, while a participant observed via video, but the Orchestra music was either linked to its movement or random. Participants rated the robots as more intelligent when the music was linked to the movement. Robots using Music Mode logged approximately two hundred hours of operation while navigating, wiping tables, and sorting trash, and bystander comments made during this operating time served as an embedded case study. This paper has both designerly contributions and engineering contributions. The contributions are: (1) an interdisciplinary choreographic, musical, and coding design process to develop a real-world robot sound feature, (2) a technical implementation for movement-based sound generation, and (3) two experiments and an embedded case study of robots running this feature during daily work activities that resulted in increased likeability and perceived intelligence of the robot.

Figures (9)

• Figure 1: Each label corresponds to a specific location on the robot, indicating the joint and corresponding musical instrument. For example, the "Gripper" corresponds to the "Percussion" sound. The robot is a prototype created by Everyday Robots.
• Figure 2: The design process for the Music Mode feature. Interviews were followed by group Brainstorming. After the first Coding Prototype was generated, we did an initial Sound Selection. This led to the Final Software that was pushed onto all 180+ robots during the Deploy on Robot stage. The elements in blue are group verbal design methods, elements in purple involve coding, and elements in green are artistic efforts.
• Figure 3: The initial off-robot signal flow implementation (A) contrasted with the final on-robot signal flow implementation (B). A. In the off-robot signal flow, a signal is captured at any layer of the robot software stack. This signal is processed per the composer's interpretation and converted in realtime into a MIDI message on a workstation. That message is consumed by a Digital Audio Workstation (DAW) carefully tuned by the composer, and a sound is generated. This system is limited by the DAW to the processing of 16 signals at once. B. In the on-robot signal flow, a signal is captured at any layer of the robot software stack. This signal is processed, triggering a predetermined sound sample. This system can process a large quantity of signals, but provides limited options for fast creative iterations. In both implementations, the signal processing included only playing sound when the velocity was above 0.05 rad/s and the velocity was over that threshold for more than 0.04 seconds
• Figure 4: The final on-robot sound implementation shown through joint velocities and produced sound volume. Joint velocity over time contrasted with the corresponding robot volume over time for the on-robot implementation. A. The torso joint with the fastest fade at 4% linear fade. The percent fade means the amount the volume was reduced every 0.04 seconds. For example, a robot speaker playing at 100% would then sound at 96% after 0.04 seconds of the joint velocity going to zero. B. The head joint with the slowest fade at 1.5%. C. The wrist with a fade of 3%.
• Figure 5: The robot performing three different tasks during Experiment 1. The top three images show the robot performing a short choreography with its arm and head. The middle three images illustrate the robot wiping a table. The bottom three images display the robot navigating across the room. The robot performed these three tasks while running each different test condition: Orchestra, Robotic, and Native.