Prim2Room: Layout-Controllable Room Mesh Generation from Primitives

TL;DR

This work proposes Prim2Room, a novel framework for controllable room mesh generation leveraging 2D layout conditions and 3D primitive retrieval to facilitate precise 3D layout specification and provides a user-friendly platform for detailed room design.

Abstract

We propose Prim2Room, a novel framework for controllable room mesh generation leveraging 2D layout conditions and 3D primitive retrieval to facilitate precise 3D layout specification. Diverging from existing methods that lack control and precision, our approach allows for detailed customization of room-scale environments. To overcome the limitations of previous methods, we introduce an adaptive viewpoint selection algorithm that allows the system to generate the furniture texture and geometry from more favorable views than predefined camera trajectories. Additionally, we employ non-rigid depth registration to ensure alignment between generated objects and their corresponding primitive while allowing for shape variations to maintain diversity. Our method not only enhances the accuracy and aesthetic appeal of generated 3D scenes but also provides a user-friendly platform for detailed room design.

Figures (6)

• Figure 1: Layout conditioned 3D room mesh generation. We propose a room generation method that takes 2D bounding boxes as input conditions. We first retrieve a 3D primitive for each object, then create the room mesh through adaptive viewpoint selection and iterative mesh generation. Our method can generate compelling textures and geometry.
• Figure 2: Generation Pipeline. Our method iteratively updates the scene mesh by a project-and-inpaint method. Before fusing the newly estimated frame into the existing mesh, we use a non-rigid depth registration method to fit the frame to both existing mesh and the conditioned primitives.
• Figure 3: Layout conditioned 3D room generation results of our method. Given 2D layout conditions, we can generate high-quality room meshes consistent with the layout specification.
• Figure 4: Qualitative Comparison of Our Method and Baseline. Comparing with Text2Room hollein2023text2room, our method can generate room meshes more consistent with the retrieved primitives. Our results also demonstrate higher-quality room boundary and furniture shapes.
• Figure 5: Ablation Study on Viewpoint Selection. Our proposed Adaptive Viewpoint Selection (AVS) algorithm helps generate contents more consistent with the retrieved primitives.