Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

"See What I Imagine, Imagine What I See": Human-AI Co-Creation System for 360$^\circ$ Panoramic Video Generation in VR

Yunge Wen

TL;DR

The paper addresses the limitation of pre-designed VR environments by introducing Imagine360, a co-creative system where AI agents assist in generating and refining 360-degree panoramic videos via speech prompts and ego-centric viewpoint adjustments. It combines Runway Gen-3-based 2D video generation with panorama-aware projection and Unity-based VR rendering, plus an AI-assisted feedback loop using Whisper-1 and GPT-3.5 Turbo. Through an eight-participant pilot comparing baseline, linear AI, and human-AI co-creation workflows, the study demonstrates that co-creation enhances creative relevance and output quality, albeit with higher cognitive load. This work presents a new paradigm for VR where users can see what they imagine and imagine what they see, paving the way for more expressive and immersive self-directed realities.

Abstract

The emerging field of panoramic video generation from text and image prompts unlocks new creative possibilities in virtual reality (VR), addressing the limitations of current immersive experiences, which are constrained by pre-designed environments that restrict user creativity. To advance this frontier, we present Imagine360, a proof-of-concept prototype that integrates co-creation principles with AI agents. This system enables refined speech-based text prompts, egocentric perspective adjustments, and real-time customization of virtual surroundings based on user perception and intent. An eight-participant pilot study comparing non-AI and linear AI-driven workflows demonstrates that Imagine360's co-creative approach effectively integrates temporal and spatial creative controls. This introduces a transformative VR paradigm, allowing users to seamlessly transition between 'seeing' and 'imagining,' thereby shaping virtual reality through the creations of their minds.

"See What I Imagine, Imagine What I See": Human-AI Co-Creation System for 360$^\circ$ Panoramic Video Generation in VR

TL;DR

The paper addresses the limitation of pre-designed VR environments by introducing Imagine360, a co-creative system where AI agents assist in generating and refining 360-degree panoramic videos via speech prompts and ego-centric viewpoint adjustments. It combines Runway Gen-3-based 2D video generation with panorama-aware projection and Unity-based VR rendering, plus an AI-assisted feedback loop using Whisper-1 and GPT-3.5 Turbo. Through an eight-participant pilot comparing baseline, linear AI, and human-AI co-creation workflows, the study demonstrates that co-creation enhances creative relevance and output quality, albeit with higher cognitive load. This work presents a new paradigm for VR where users can see what they imagine and imagine what they see, paving the way for more expressive and immersive self-directed realities.

Abstract

The emerging field of panoramic video generation from text and image prompts unlocks new creative possibilities in virtual reality (VR), addressing the limitations of current immersive experiences, which are constrained by pre-designed environments that restrict user creativity. To advance this frontier, we present Imagine360, a proof-of-concept prototype that integrates co-creation principles with AI agents. This system enables refined speech-based text prompts, egocentric perspective adjustments, and real-time customization of virtual surroundings based on user perception and intent. An eight-participant pilot study comparing non-AI and linear AI-driven workflows demonstrates that Imagine360's co-creative approach effectively integrates temporal and spatial creative controls. This introduces a transformative VR paradigm, allowing users to seamlessly transition between 'seeing' and 'imagining,' thereby shaping virtual reality through the creations of their minds.
Paper Structure (10 sections, 2 figures)

This paper contains 10 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Related Works
  3. The Emerging Technology of Panoramic Video Generation
  4. Human-AI Co-Creation in Immersive Environment
  5. System Design
  6. Pilot Study
  7. Parameters and Procedure
  8. Results
  9. Discussion and Future Work
  10. Conclusion

Figures (2)

  • Figure 1: Co-Creation Workflow of Panorama Video Generation in VR. The system enables users to (1) generate panoramic videos with AI assistance, (2) evaluate generation outcomes in real time, (3) provide speech-based text prompts that are refined and supported by the AI agent, and (4) recenter the panorama video’s focal point based on an egocentric perspective. This co-creation workflow leverages AI's capabilities in the backend and offers suggestions upon inquiry that seamlessly connect the user's intent with their perception.
  • Figure 2: Qualitative Comparisons of Participant Generation Outcomes. Participants reported a preference for (b) iteratively collaborating with agents to refine generation outcomes and enhance creative potential, compared to (a) passively receiving generation results without agent interaction.