Nudging the Somas: Exploring How Live-Configurable Mixed Reality Objects Shape Open-Ended Intercorporeal Movements

TL;DR

GravField, a research-through-design, co-located MR performance system where an object jockey live-configures virtual objects with real-time, parameterized digital physics to influence headset-wearing participants'movement, offers empirical insights and design principles for MR systems that can guide group movement through embodied, felt dynamics while preserving participants'sense of agency.

Abstract

Mixed Reality (MR) increasingly explores how virtual elements can shape physical behavior, yet how MR objects guide group movement remains underexplored. We address this gap by examining how virtual objects can nudge collective, co-located movement without relying on explicit instructions or choreography. We developed GravField, a research-through-design, co-located MR performance system where an "object jockey" live-configures virtual objects (e.g., ropes, springs, magnetic fields) with real-time, parameterized "digital physics" (e.g., weight, elasticity, force) to influence headset-wearing participants' movement, made perceptible through augmented visual and audio feedback serving as cognitive-somatic cues. Our bricolage analysis of the performances, based on video, interviews, soma trajectories, and field notes, indicates that these live nudges support emergent intercorporeal coordination and that ambiguity and real-time configuration sustain open-ended, exploratory engagement. Ultimately, our work offers empirical insights and design principles for MR systems that can guide group movement through embodied, felt dynamics while preserving participants' sense of agency.

Figures (14)

• Figure 1: System architecture of GravField. HoloKit X headsets stream six-degrees-of-freedom pose data to a Unity server, which computes Mixed Reality Object physics, renders visual effects back to headsets, and sends Open Sound Control messages to Ableton Live for real-time audio synthesis. A spectator iPad and TV display the mixed reality overlay for audience viewing.
• Figure 2: OJ1's TouchOSC control panel for real-time configuration of "digital physics." Three mode buttons (top) switch between the Rope, Spring, and Magnetic MROs, each exposing mode-specific parameter sliders. The OJ adjusts these on-the-fly during a performance to nudge collective movement.
• Figure 3: OJ2's Unity control interface for monitoring participants' real-time state. Left panel: incoming OSC parameters including interpersonal distances, heights, rope velocity and acceleration, and magnetic field values. Background: the simulated scene rendering three participants connected by Rope MROs with real-time data streaming.
• Figure 4: The "Rope" Mixed Reality Object. Left: rope velocity ($v$) maps to audio volume and centripetal acceleration ($a$) to pitch, rewarding faster continuous motion. Right: participants swinging the virtual rope during a performance.
• Figure 5: The "Spring" Mixed Reality Object. Left: horizontal distance ($d$) controls pitch and relative height ($h$) controls a low-pass filter, encouraging push--pull and see-saw dynamics. Right: participants exploring vertical and lateral movements during a performance.