Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Immersive Mixed Reality Street Play: Understanding Co-located Bodily Play with See-through Head-mounted Displays in Public Spaces

Botao Amber Hu, Rem Rungu Lin, Yilan Elan Tao, Samuli Laato, Yue Li

TL;DR

This paper investigates the social implications of Immersive Mixed Reality Street Play (IMRSP) by deploying MOFA, an open-source research-through-design framework, in real-world urban settings with see-through MRHMDs. It documents how co-located, movement-based play reconfigures social dynamics, spectator roles, and public space usage, while revealing social, ethical, and safety barriers to broader adoption. The study advances design recommendations and a methodological agenda for IMRSP, including contextual awareness, explainable social affordances, and safety-focused interaction paradigms, and introduces a low-cost, offline co-location approach via InstantCopresence and Multipeer Connectivity. These insights inform both future research and practical guidance for deploying socially acceptable public MR experiences, balancing engagement with public norms and safety. Overall, IMRSP is feasible today but requires addressing normative, privacy, and physical-safety concerns to achieve widespread, responsible adoption in urban life.

Abstract

As see-through Mixed Reality Head-Mounted Displays (MRHMDs) proliferate, their usage is gradually shifting from controlled, private settings to spontaneous, public contexts. While location-based augmented reality mobile games such as Pokemon GO have been successful, the embodied interaction afforded by MRHMDs moves play beyond phone-based screen-tapping toward co-located, bodily, movement-based play. In anticipation of widespread MRHMD adoption, major technology companies have teased concept videos envisioning urban streets as vast mixed reality playgrounds-imagine Harry Potter-style wizard duels in city streets-which we term Immersive Mixed Reality Street Play (IMRSP). However, few real-world studies examine such scenarios. Through empirical, in-the-wild studies of our research-through-design game probe, Multiplayer Omnipresent Fighting Arena (MOFA), deployed across diverse public venues, we offer initial insights into the social implications, challenges, opportunities, and design recommendations of IMRSP. The MOFA framework, which includes three gameplay modes-"The Training," "The Duel," and "The Dragon"-is open-sourced at https://github.com/realitydeslab/mofa.

Towards Immersive Mixed Reality Street Play: Understanding Co-located Bodily Play with See-through Head-mounted Displays in Public Spaces

TL;DR

This paper investigates the social implications of Immersive Mixed Reality Street Play (IMRSP) by deploying MOFA, an open-source research-through-design framework, in real-world urban settings with see-through MRHMDs. It documents how co-located, movement-based play reconfigures social dynamics, spectator roles, and public space usage, while revealing social, ethical, and safety barriers to broader adoption. The study advances design recommendations and a methodological agenda for IMRSP, including contextual awareness, explainable social affordances, and safety-focused interaction paradigms, and introduces a low-cost, offline co-location approach via InstantCopresence and Multipeer Connectivity. These insights inform both future research and practical guidance for deploying socially acceptable public MR experiences, balancing engagement with public norms and safety. Overall, IMRSP is feasible today but requires addressing normative, privacy, and physical-safety concerns to achieve widespread, responsible adoption in urban life.

Abstract

As see-through Mixed Reality Head-Mounted Displays (MRHMDs) proliferate, their usage is gradually shifting from controlled, private settings to spontaneous, public contexts. While location-based augmented reality mobile games such as Pokemon GO have been successful, the embodied interaction afforded by MRHMDs moves play beyond phone-based screen-tapping toward co-located, bodily, movement-based play. In anticipation of widespread MRHMD adoption, major technology companies have teased concept videos envisioning urban streets as vast mixed reality playgrounds-imagine Harry Potter-style wizard duels in city streets-which we term Immersive Mixed Reality Street Play (IMRSP). However, few real-world studies examine such scenarios. Through empirical, in-the-wild studies of our research-through-design game probe, Multiplayer Omnipresent Fighting Arena (MOFA), deployed across diverse public venues, we offer initial insights into the social implications, challenges, opportunities, and design recommendations of IMRSP. The MOFA framework, which includes three gameplay modes-"The Training," "The Duel," and "The Dragon"-is open-sourced at https://github.com/realitydeslab/mofa.
Paper Structure (87 sections, 8 figures, 1 table)

This paper contains 87 sections, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Technology-mediated Street Play
  4. Technology-supported Co-located Bodily Play
  5. Co-located Mixed Reality
  6. Mixed Reality Concerns in Public Spaces
  7. Imbalanced power dynamics
  8. Embodied interactions
  9. Social acceptance and safety in public spaces
  10. Research Gap: Immersive Mixed Reality Street Play
  11. Inaccessibility of affordable MRHMDs
  12. Limited technology for spontaneous, low‑latency multiplayer mixed reality co-location
  13. Social acceptance barriers and lack of established social norms
  14. Methods for Exploring Future Mixed Reality
  15. System Design
  16. ...and 72 more sections

Figures (8)

  • Figure 1: Detailed game and user experience design, using "The Duel" as an example. It illustrates: (1) the first-person perspective of the game with all UI elements users can view through the MR interface; (2) interaction design details using Apple Watch to detect spell-casting gestures; (3) magic spell and shield system design; (4) co-location and networking user flow: how two players connect and how spectators join to watch in third-person view; (5) detailed round-based dueling game mechanism design; and (6) conversion process from monocular screen-based AR view into immersive stereoscopic MR view based on HoloKit.
  • Figure 2: Top: Device-pairing diagram for MOFA "The Dragon". The image on the right shows a frame from a spectator video recorded on Device 6. Bottom: Step-by-step onboarding process: (a) All participants launch the game app on their iPhones. (b) The host device initiates the game session, and guest devices join by scanning a QR code displayed on the host’s screen. (c) Wizard players switch to stereoscopic mixed reality mode and insert their phones into the headsets. (d) Wizard players press the "Ready" button on their watches to activate their powers and enter the game. (e) After each round, performance data---"Score" (kills), "Hit Rate", "Distance" (distance moved), and "Calorie" (calories burned)---are displayed on each wizard player's Apple Watch. Players can tap "Play Again" on the watch to start a new round.
  • Figure 3: Spectators can use iPads to view the augmented reality experience from a third-person perspective and capture recordings of what they see.
  • Figure 4: Five representative scenarios of volunteered participation and interviews. S1: Campus; S2: Expos; S3: Urban Parks; S4: Public Buildings; S5: Natural Wilderness.
  • Figure 5: Data collection from the in-the-wild study.
  • ...and 3 more figures