GenEx: Generating an Explorable World

TL;DR

GenEx addresses the challenge of enabling AI to understand and explore 3D environments by generating a physically grounded, 360° explorable world from a single image. It combines a diffusion-based world initializer with an action-driven panoramic transition, all trained on physics-engine data, and enlists imaginative agents augmented by GPTs to perform goal-agnostic and goal-driven tasks. A key contribution is the imagination-augmented policy, including a multi-agent variant, which leverages imagined observations to improve decision making and planning. The results demonstrate high-quality, 3D-consistent generation, long-range loop stability, BEV generation, active 3D mapping, and improved embodied decisions, indicating substantial potential for real-world navigation, immersive gaming, and VR/AR applications.

Abstract

Understanding, navigating, and exploring the 3D physical real world has long been a central challenge in the development of artificial intelligence. In this work, we take a step toward this goal by introducing GenEx, a system capable of planning complex embodied world exploration, guided by its generative imagination that forms priors (expectations) about the surrounding environments. GenEx generates an entire 3D-consistent imaginative environment from as little as a single RGB image, bringing it to life through panoramic video streams. Leveraging scalable 3D world data curated from Unreal Engine, our generative model is rounded in the physical world. It captures a continuous 360-degree environment with little effort, offering a boundless landscape for AI agents to explore and interact with. GenEx achieves high-quality world generation, robust loop consistency over long trajectories, and demonstrates strong 3D capabilities such as consistency and active 3D mapping. Powered by generative imagination of the world, GPT-assisted agents are equipped to perform complex embodied tasks, including both goal-agnostic exploration and goal-driven navigation. These agents utilize predictive expectation regarding unseen parts of the physical world to refine their beliefs, simulate different outcomes based on potential decisions, and make more informed choices. In summary, we demonstrate that GenEx provides a transformative platform for advancing embodied AI in imaginative spaces and brings potential for extending these capabilities to real-world exploration.

Figures (12)

• Figure 1: Our data curation leverages physical engines, utilizing realistic city assets from UE5 and animated world assets from Unity.
• Figure 2: Three panorama representations that can be transformed into one another.
• Figure 3: From single view to 360$^\circ$ panorama.
• Figure 4: We model the world transition as a panoramic video generation process. Given the last explored 360$^{\circ}$ panorama and an action that rotates the viewing sphere, the model produces a sequence of newly generated panoramic views
• Figure 5: Three exploration modes --- interactive, GPT-assisted, and goal-driven --- each defined by distinct exploration instructions.

Theorems & Definitions (4)

• Definition D.1: Spherical polar coordinate system
• Definition D.2: Cartesian coordinate system for panoramic image
• Definition D.3: Sphere-to-Cartesian Coordinate Transformation
• Definition D.4: Rotation Transformation in Spherical Polar Coordinate System