LPA3D: 3D Room-Level Scene Generation from In-the-Wild Images

TL;DR

LPA3D tackles the problem of generating semantically coherent room-level 3D indoor scenes from in-the-wild RGB images with unknown poses. It introduces Local-Pose-Alignment (LPA), an anchor-based multi-local-coordinate system, and a NeRF-based GAN (LPA-GAN) that jointly optimizes a local pose predictor and the 3D scene generator, using boundary-aware room-size conditioning and view-based camera sampling. The approach achieves superior inter-view consistency and reduced layout abnormalities compared to state-of-the-art object- and diffusion-based methods, and demonstrates scalability when trained on large in-the-wild image collections. Limitations include the box-like room assumption and exclusive 2D RGB supervision, suggesting future integration of depth priors and global layout cues to further improve realism and geometry.

Abstract

Generating realistic, room-level indoor scenes with semantically plausible and detailed appearances from in-the-wild images is crucial for various applications in VR, AR, and robotics. The success of NeRF-based generative methods indicates a promising direction to address this challenge. However, unlike their success at the object level, existing scene-level generative methods require additional information, such as multiple views, depth images, or semantic guidance, rather than relying solely on RGB images. This is because NeRF-based methods necessitate prior knowledge of camera poses, which is challenging to approximate for indoor scenes due to the complexity of defining alignment and the difficulty of globally estimating poses from a single image, given the unseen parts behind the camera. To address this challenge, we redefine global poses within the framework of Local-Pose-Alignment (LPA) -- an anchor-based multi-local-coordinate system that uses a selected number of anchors as the roots of these coordinates. Building on this foundation, we introduce LPA-GAN, a novel NeRF-based generative approach that incorporates specific modifications to estimate the priors of camera poses under LPA. It also co-optimizes the pose predictor and scene generation processes. Our ablation study and comparisons with straightforward extensions of NeRF-based object generative methods demonstrate the effectiveness of our approach. Furthermore, visual comparisons with other techniques reveal that our method achieves superior view-to-view consistency and semantic normality.

Figures (10)

• Figure 1: The workflow of LPA3D involves a 3D room-level indoor scene generative model trained on millions of in-the-wild images with unknown global poses. This training is facilitated by local-pose-alignment, a novel pose representation that categorizes in-the-wild images into anchor-based local pose coordinate systems. Subsequently, LPA-GAN, a generative model based on neural radiance fields (NeRFs) and adapted to LPA, enables the generation of room-level scenes. It supports 360-degree panoramas and free navigation, leading to a wide range of applications.
• Figure 2: Process of anchor identification and establishing a local coordinate system based on the corners of a 'core object' $c$, such as a bed in a bedroom. Given an image, the front ($F$) and right-side ($R$) directions of $c$, are determined. The corner is then identified, and its ID is queried based on predefined criteria, like $a_1$, located at the left side of the bed's head as shown in the floorplan. Finally, a local right-hand coordinate system is constructed with axes $\{\bm{x}, \bm{y}, \bm{z}\}$.
• Figure 3: Overview of LPA-GAN. We adopt a co-optimization strategy to iteratively train the generative model and camera predictor. During the training iterations of the generative model, highlighted by green lines, only the generator ($G$) and discriminator ($D$) are trainable. The camera predictor ($C$), which remains frozen during this phase, estimates camera poses from real images, providing these poses as candidates for subsequent sampling processes. The generative sampler for rendering ($GSR$) then selects a camera pose within the specified synthetic scene to render a synthetic view. In contrast, during the training iterations of the camera predictor, depicted by orange lines, the predictive sampler for rendering ($PSR$) uniformly samples views within synthetic scenes generated by the now-frozen generator. $PSR$ supplies rendered images and their corresponding camera poses as training samples for training the camera predictor.
• Figure 4: Various 3D scenes generated by our model. The room types, arranged from top to bottom, are bedroom, living room, and kitchen. Each scene is depicted as a panoramic view and two perspective views, including both RGB and depth images.
• Figure 5: Qualitative comparisons with existing indoor scene generation methods. Each row contains two scenes, each rendered from three different views. The view-inconsistent 3D contents and abnormal layout arrangement are highlighted using red boxes.