Storycaster: An AI System for Immersive Room-Based Storytelling

TL;DR

Storycaster introduces an immersive room-based storytelling system that uses spatial augmented reality and generative AI to transform a real room into a dynamic narrative world without wearing a headset. A Narrator Agent orchestrates multiple GPT-4.1 components to generate scene scripts, visuals, ambient audio, and character dialogue, while enabling object-level editing of room objects. The system employs a cylindrical image-generation pipeline with Depth ControlNet and 360° LoRA, plus a Model Context Protocol to coordinate server-side generation and playback, delivering three-act stories guided by user voice input. In a user study with thirteen participants, narrator guidance and spatial audio were key to immersion, though latency and visual fidelity emerged as important areas for improvement. The work demonstrates a pathway toward open-ended, embodied storytelling that blends physical space with AI-generated content, with broad implications for entertainment, education, and well-being.

Abstract

While Cave Automatic Virtual Environment (CAVE) systems have long enabled room-scale virtual reality and various kinds of interactivity, their content has largely remained predetermined. We present \textit{Storycaster}, a generative AI CAVE system that transforms physical rooms into responsive storytelling environments. Unlike headset-based VR, \textit{Storycaster} preserves spatial awareness, using live camera feeds to augment the walls with cylindrical projections, allowing users to create worlds that blend with their physical surroundings. Additionally, our system enables object-level editing, where physical items in the room can be transformed to their virtual counterparts in a story. A narrator agent guides participants, enabling them to co-create stories that evolve in response to voice commands, with each scene enhanced by generated ambient audio, dialogue, and imagery. Participants in our study ($n=13$) found the system highly immersive and engaging, with narrator and audio most impactful, while also highlighting areas for improvement in latency and image resolution.

Figures (14)

• Figure 1: System overview of Storycaster. The Narrator orchestrates the storytelling process, prompting the user for input and guiding them through three acts. The Scene Converser gathers initial story elements, which are passed to the Scene Creator. Within the Scene Creator, the Storyteller generates narrative text, the Image Generator creates visuals, the Virtual Object Mapper overlays virtual counterparts onto physical objects via inpainting, and the Scene Audio Narrator produces ambient audio and character dialogue. The Story Coach manages transitions between acts, allowing users to either continue the narrative or change objects in the room. Segmentation, inpainting, ambient audio generation, and image generation all happen on the server side. This process repeats for three acts, after which the user can choose to begin another story or conclude the session.
• Figure 2: Storycaster combines multiple color and depth cameras to generate visuals that follow the physical features of the room. Calibrated color (a) and depth cameras (b) (rendered as false color) are combined to create a textured mesh that serves as a realtime geometric model of the room (c). This model is rendered to a single cube map (d). The cube map z-buffer is re-rendered as a cylindrical (or 360°) depth image (e). To render an exterior or outdoor scene, walls may be optionally removed (f). Stable diffusion is applied with a depth ControlNet conditioned on (f) and a 360° LoRA model. ControlNet effects are masked (eliminated) for pixels that are zero in (f), allowing for a more natural horizon effect in exterior (outdoor) settings. The generated image (g) is upscaled 4x (not shown). The image displayed at each projector is calculated from the geometric model of the room, using intrinsic and extrinsic parameters of the projectors (h). Black pixels correspond to regions of the geometry that are not visible in (e) (i.e., they are occluded by another object). The final projected visuals create 360° views into an exterior virtual space and change the appearance of objects in the room (i).
• Figure 3: Examples of object editing from user studies. Participants selected real-world objects (table and ottoman, shown with their masks on the left) and reimagined them as virtual objects within their stories. The resulting generations (right) illustrate diverse mappings such as a campfire, quilt, black hole, burrito, and flowers, demonstrating the flexibility of object-level transformations in Storycaster.
• Figure 4: Examples of using Grounded SAM to identify virtual objects by passing in object prompts with a lowered similarity threshold (0.1). Top row: prompting “boat” highlights the sofa as the closest visual match. Bottom row: prompting “vine” highlights the ladder. From left to right: original room image, all detected masks, and the best detection mask. This demonstrates the ability to flexibly map real objects to virtual counterparts, enabling novel object-level editing in immersive storytelling.
• Figure 5: Generations from user studies ($n=13$). Each row represents one participant, with three images corresponding to the acts of their story. Labels indicate participant ID and act number (e.g., P51 = Participant 5, Act 1). Images marked with (N) denote acts where the user allowed the Narrator to decide the direction of the story. The examples highlight the diversity of user-authored narratives, ranging from historical explorations and mythological tales to domestic scenes, space adventures, and fantasy worlds, demonstrating the flexibility of Storycaster in supporting a wide variety of storytelling themes.