Creativity and Visual Communication from Machine to Musician: Sharing a Score through a Robotic Camera

TL;DR

This paper explores the integration of visual communication and musical interaction by implementing a robotic camera within a"Guided Harmony" musical game and contributes to the broader understanding of machine intelligence in augmenting human creativity, particularly in musical settings.

Abstract

This paper explores the integration of visual communication and musical interaction by implementing a robotic camera within a "Guided Harmony" musical game. We aim to examine co-creative behaviors between human musicians and robotic systems. Our research explores existing methodologies like improvisational game pieces and extends these concepts to include robotic participation using a PTZ camera. The robotic system interprets and responds to nonverbal cues from musicians, creating a collaborative and adaptive musical experience. This initial case study underscores the importance of intuitive visual communication channels. We also propose future research directions, including parameters for refining the visual cue toolkit and data collection methods to understand human-machine co-creativity further. Our findings contribute to the broader understanding of machine intelligence in augmenting human creativity, particularly in musical settings.

Figures (5)

• Figure 1: Jacob Collier leads a performance of "Conducting the Audience". After providing initial pitches to two sections of the audience, he uses his hands to indicate to these sections to move their pitch up or down a step in a live performance.
• Figure 2: (Top) The ImproVision robot turns to make pseudo "eye contact" with the violinist. After turning to indicate who ImproVision is communicating to, it will then provide an instruction through its motion. (Bottom) The process of providing an ensemble down-beat: the ImproVision robot first looks to ensemble center, lifts, then bows its head.
• Figure 3: A high-level diagram of the ImproVision system. Based on a score, the machine initializes pitches for the musicians. The system observes the musicians, maintaining the harmony unless a Musician Request Signal (in this case, a nonverbal signal) is detected. If a Musician Request Signal is detected, the machine communicates the desired new pitches through movements. If the end of the score is reached, the machine conveys an end-of-piece signal. Note that the score is not necessarily fixed; it could merely be a generative reference throughout.
• Figure 4: The clarinet player provides a signal to the ImproVision system that he is ready to proceed further in the score. The ImproVision system will observe the signal and then provide new instructions to the ensemble.
• Figure 5: The ImproVision system maintains awareness of the state of each musician; in this setting, the system detects pose keypoints of the musicians in order to observe the predefined nonverbal cue (hand-raise).