Beyond Reality: Designing Personal Experiences and Interactive Narratives in AR Theater

TL;DR

This dissertation advances AR-based immersive theater by introducing Dynamic Theater, Spatial Orchestra, and Audience Amplified as a cohesive platform for walkable, location-based narratives. It combines wide-area AR, digital twin pipelines, and AI-trained virtual audiences to enable anytime, anywhere narrative experiences that preserve the social dynamics of live performance. A series of outdoor and indoor studies evaluate lighting, navigation aids, and audience presence, showing that guided space design and AI audiences can enhance engagement and perceived realism while maintaining user autonomy. The work culminates in an open-source AR theater platform and a teaching module, offering practical guidelines for artists and researchers to design collaborative, scalable, and personalized AR performances.

Abstract

Augmented Reality (AR) technologies are redefining how we perceive and interact with the world by seamlessly integrating digital elements into our physical surroundings. These technologies offer personalized experiences and transform familiar spaces by layering new narratives onto the real world. Through increased levels of perceived agency and immersive environments, my work aims to merge the human elements of live theater with the dynamic potential of virtual entities and AI agents. This approach captures the subtlety and magic of storytelling, making theater experiences available anytime and anywhere. The system I am building introduces innovative methods for theatrical production in virtual settings, informed by my research and eight published works. These contributions highlight domain-specific insights that have shaped the design of an immersive AR Theater system. My research in building a well-designed AR stage features avatars and interactive elements that allow users to engage with stories at their own pace, granting them full agency over their experience. However, to ensure a smooth and curated experience that aligns with the director or creator's vision, several factors must be considered, especially in open-world settings that depend on natural user movement. This requires the story to be conveyed in a controlled manner, while the interaction remains intuitive and natural for the user.

Figures (48)

• Figure 1.1: An infographic describing the different areas of study included in this dissertation and their relationship to certain developments in the field of HCI. The subsequent chapters detail studies conducted to target areas of possible development in these categories. Related, EEG, haptic data, and eye gaze strengthen understandings for AI agents and audience participation. Combined, these studies contribute to greater corroboration between performance and extended reality.
• Figure 1.2: This graphic illustrates the selected works featured in this dissertation as they were developed in graduate school. It also organizes the projects based on their association with more artistic works, such as Spatial Orchestra (Chapter 4), or more engineering-focused works, such as AR Outdoor Navigation Aids (Chapter 3).
• Figure 1.3: This Venn Diagram illustrates the main fields of study I engaged throughout my research and the areas of overlap that I exploited in my projects. It is in these areas of overlap that this dissertation's vision comes to fruition and makes the systems I propose viable.
• Figure 2.4: (a) The popular VR game Road 96 was initially released in 2021; (b) The Under Presents released in 2019, included a live performance component.
• Figure 2.5: Immersive Theater Works. (a) Morel’s Panorama by Masaki Fujihata. (b) Variations V by Merce Cunningham. (c) BeHere 1942 - Masaki Fujihata. (d) Boundary Functions by Theodore John Kaczynski. (e) Pattern Recognition by Alexander Whitley. (f) Pixel by Mourad Merzouki. (g) Solo by Deborah Hay. (h) Centaur by Pontus Lidberg.