Spatial Orchestra: Locomotion Music Instruments through Spatial Exploration

TL;DR

Spatial Orchestra presents an AR musical instrument that enables music creation through natural locomotion within a 3.3 m × 3.3 m fenced play space. Ten color-coded bubbles hover at head height and emit synthesized cello notes drawn from the chord set $[EMaj, Em, FMaj7, GMaj, G7, Am, Bdim, Bm5, Cmaj, Dm]$, with sound localized to the user’s position and center omitted. The system uses a standalone AR headset and spatial audio to map movement into spatially localized notes as users walk into bubbles, enabling immediate participation without prior training. Iterative design refinements—improved bubble rendering, inner cues, and height-adjustable accessibility modes—alongside safety features and Azure Spatial Anchors reliability, demonstrate feasibility and inclusivity of embodied MR music experiences. Overall, the work demonstrates how spatial interaction and musical embodiment can broaden participation in music-making, with practical implications for education and public demonstrations.

Abstract

Spatial Orchestra demonstrates how easy it is to play musical instruments using basic input like natural locomotion, which is accessible to most. Unlike many musical instruments, our work allows individuals of all skill levels to effortlessly create music by walking into virtual bubbles. Our Augmented Reality experience involves interacting with ever-shifting sound bubbles that the user engages with by stepping into color-coded bubbles within the assigned area using a standalone AR headset. Each bubble corresponds to a cello note, and omits sound from the center of the bubble, and lets the user hear and express in spatial audio, effectively transforming participants into musicians. This interactive element enables users to explore the intersection of spatial awareness, musical rhythm that extends to bodily expression through playful movements and dance-like gestures within the bubble-filled environment. This unique experience illuminates the intricate relationship between spatial awareness and the art of musical performance.

Figures (3)

• Figure 1: An aerial perspective of the stage arrangement that the user explores. a. The physical space used for the Spatial Orchestra. b. The fencing and virtual stage are all augmented. c. There are ten augmented virtual bubbles set up in the area. Each bubble emits unique cello notes that are color coded to enable user interaction.
• Figure 2: a. The bubble rendering model underwent improvements based on user feedback. Left: The original bubble featured a color highlight at the edge, representing the bubble's assigned color. Right: The original bubble used had a color highlight of the bubble’s assigned color at the edge of the bubble. Right: A new shader incorporated a white highlight for improved visibility of the bubble's boundaries. Adjustments were also made to the transparency and illumination levels, addressing user concerns about clarity when multiple bubbles are stacked and colors are mixed. b. We addressed a critical issue with the original shader – users struggled to identify if they were inside the bubble. To solve this, we added a visual cue. Now, when users enter the bubble, the scene adopts a translucent color within the bubble's view. This was achieved through a double-sided shader, rendering both sides of the mesh.
• Figure 3: a. Compares the virtual bubbles' size to the virtual fence's height. Based on individual viewpoint heights, it may be challenging for some people to interact with bubbles directly. b. View from the person in a wheelchair engaging with the bubbles. c. We implemented an accessibility mode that allowed manual adjustment to the height of the bubbles. As you can see, the bubbles appear at the correct height when viewed from a wheelchair.