NuiWorld: Exploring a Scalable Framework for End-to-End Controllable World Generation

TL;DR

NuiWorld tackles the twin challenges of scalability and data scarcity in open-domain world generation by coupling a generative bootstrapping pipeline with a scalable, end-to-end sketch-conditioned model. It builds on NuiScene’s chunk-based vector-set representation and quad chunk diffusion to maintain fidelity across scene sizes, while introducing a training pipeline that bootstraps from a few input images to synthesize diverse 3D scenes. A sketch-to-world model conditioned on pseudo sketches enables end-to-end controllable generation, with a size-prediction module to adapt token counts to scene layout. The approach demonstrates improved scalability and controllability over prior training-free and voxel-based methods, though it acknowledges limitations in generalization and final scene coherence that invite further scaling and decomposition work.

Abstract

World generation is a fundamental capability for applications like video games, simulation, and robotics. However, existing approaches face three main obstacles: controllability, scalability, and efficiency. End-to-end scene generation models have been limited by data scarcity. While object-centric generation approaches rely on fixed resolution representations, degrading fidelity for larger scenes. Training-free approaches, while flexible, are often slow and computationally expensive at inference time. We present NuiWorld, a framework that attempts to address these challenges. To overcome data scarcity, we propose a generative bootstrapping strategy that starts from a few input images. Leveraging recent 3D reconstruction and expandable scene generation techniques, we synthesize scenes of varying sizes and layouts, producing enough data to train an end-to-end model. Furthermore, our framework enables controllability through pseudo sketch labels, and demonstrates a degree of generalization to previously unseen sketches. Our approach represents scenes as a collection of variable scene chunks, which are compressed into a flattened vector-set representation. This significantly reduces the token length for large scenes, enabling consistent geometric fidelity across scenes sizes while improving training and inference efficiency.

Figures (18)

• Figure 1: Object generators like Trellis 2 do not adapt to scene scale or aspect ratio, causing such scenes to be represented with less voxels and degraded fidelity while our method scales with scene sizes and maintains fidelity.
• Figure 2: Our framework begins with generative bootstrapping, shown on the left. Using Nano Banana to generate images and Trellis 2 to reconstruct 3D scenes. NuiScene is then trained on these scenes to produce new scenes with varying size and layouts. Finally, using these scenes and their pseudo sketches we train our variable-length sketch-to-world model on the right.
• Figure 3: Qualitative comparison of ground-truth NuiScene scenes and generated outputs from sketch-to-world models on the medieval overfit set across different VecSet compression ratios and model widths.
• Figure 4: Qualitative comparison with Trellis 2. Trellis 2 takes as input a rendered view of the scene generated by our sketch-to-world model (top left), which itself is conditioned on the sketch shown in the bottom left. To better utilize the $1024\times1024$ input resolution for Trellis 2 and minimize detail loss for elongated scenes, we rotate the scene during rendering. The center column shows the fully generated scenes, while the right column presents zoomed-in renderings.
• Figure 5: The top row shows a large medieval scene image and its corresponding sketch both generated by Nano Banana. The bottom row presents the results produced by Trellis 2 and our XL model using the above as input. Zoom in for more details.