From Score to Sound: An End-to-End MIDI-to-Motion Pipeline for Robotic Cello Performance

TL;DR

This work presents an end-to-end MIDI to motion pipeline that converts cello MIDI scores into collision-aware bowing trajectories for a UR5e robot, enabling sight-reading without motion capture. It validates perceptual indistinguishability to human performance via a Musical Turing Test with 132 participants and provides a synchronized dataset of robot and human performances for five beginner pieces. The approach demonstrates that non-musicians struggle to reliably distinguish robot from human cello play, suggesting the method yields musically convincing robotic performances and offering a reproducible benchmark for robotic musicianship research. The paper also discusses limitations and outlines future directions including force sensing, auditory feedback, and learning-based refinements to improve string crossing and sound quality.

Abstract

Robot musicians require precise control to obtain proper note accuracy, sound quality, and musical expression. Performance of string instruments, such as violin and cello, presents a significant challenge due to the precise control required over bow angle and pressure to produce the desired sound. While prior robotic cellists focus on accurate bowing trajectories, these works often rely on expensive motion capture techniques, and fail to sightread music in a human-like way. We propose a novel end-to-end MIDI score to robotic motion pipeline which converts musical input directly into collision-aware bowing motions for a UR5e robot cellist. Through use of Universal Robot Freedrive feature, our robotic musician can achieve human-like sound without the need for motion capture. Additionally, this work records live joint data via Real-Time Data Exchange (RTDE) as the robot plays, providing labeled robotic playing data from a collection of five standard pieces to the research community. To demonstrate the effectiveness of our method in comparison to human performers, we introduce the Musical Turing Test, in which a collection of 132 human participants evaluate our robot's performance against a human baseline. Human reference recordings are also released, enabling direct comparison for future studies. This evaluation technique establishes the first benchmark for robotic cello performance. Finally, we outline a residual reinforcement learning methodology to improve upon baseline robotic controls, highlighting future opportunities for improved string-crossing efficiency and sound quality.

Figures (7)

• Figure 1: Experimental setup of the robotic cellist, including the UR5e robotic arm, cello, custom stand, and metal frame.
• Figure 2: MIDI-to-Motion pipeline which converts a musical sequence into robotic motions, allowing for capture of demonstration data.
• Figure 3: Method for translating a MIDI music score into a parameterized note sequence with attached information on note name, note duration, cello string, and bowing.
• Figure 4: Illustration of robotic bowing primitives and string crossings. Left: example of a string crossing movement, showing projection of the current bow fraction onto the target string. Right: example of frog ($F_s$) and tip ($T_s$) waypoints with the corresponding bow length $\lVert T_s - F_s \rVert$. Note: UR5e robot image is adapted from ur5e-image.
• Figure 5: System setup for robotic cello playing. (Left) UR5e robot arm positioned with a cello in MuJoCo simulation, (Middle) CAD design of the instrument support and base platform, (Right) custom-designed bow gripper for precise bowing control.