AURORA: Automated Unleash of 3D Room Outlines for VR Applications

TL;DR

AURORA tackles the labor-intensive challenge of VR interior design by converting RGB-D sequences into accurate 3D interior outlines suitable for fully virtual or hybrid scenes that integrate CAD assets. The pipeline combines GS-based SLAM (via SplaTAM) with geometry-refined surface reconstruction using two novel losses, followed by TSDF-fusion conversion, 3D instance segmentation with SoftGroup++, envelope extraction, and ShapeNet-based model registration, all guided by interior priors. Key contributions include the introduction of $\ ext{L}_{normal}$ and $\

Abstract

Creating realistic VR experiences is challenging due to the labor-intensive process of accurately replicating real-world details into virtual scenes, highlighting the need for automated methods that maintain spatial accuracy and provide design flexibility. In this paper, we propose AURORA, a novel method that leverages RGB-D images to automatically generate both purely virtual reality (VR) scenes and VR scenes combined with real-world elements. This approach can benefit designers by streamlining the process of converting real-world details into virtual scenes. AURORA integrates advanced techniques in image processing, segmentation, and 3D reconstruction to efficiently create realistic and detailed interior designs from real-world environments. The design of this integration ensures optimal performance and precision, addressing key challenges in automated indoor design generation by uniquely combining and leveraging the strengths of foundation models. We demonstrate the effectiveness of our approach through experiments, both on self-captured data and public datasets, showcasing its potential to enhance virtual reality (VR) applications by providing interior designs that conform to real-world positioning.

Figures (3)

• Figure 1: This illustration demonstrates our pipeline, where user-provided RGB-D images are processed to generate multiple room outline options. The first option, named virtual outline, replaces the entire scene with CAD models for a fully structured representation. The second option, named hybrid outline, replaces selected furniture items with CAD models while retaining other elements from the 3D reconstruction for a blended representation. The input images are processed via a Gaussian Splatting-based module, generating 3D Gaussians with camera poses (top middle), then refined with normal priors to improve accuracy and detail (top right). This step is followed by a conversion from GS to a textured point cloud for feeding into the instance segmentation module. The 3D segmentation module then identifies and separates walls, floors, and furniture. Segmented walls and floors (bottom middle) are used to extract an actual-size floor with surrounding walls, while segmented furniture (bottom middle) is matched to database models, presenting outline options to the user.
• Figure 2: Results of the model placement in the 3D scene. We replace the segmented point cloud with the candidate models from the ShapeNet chang2015shapenet to explore various interior designs. Our model could randomly generate multiple model placements in the scene. We show three of them for qualitative evaluation. (a) is the segmented point cloud of the interested object. (b-d) show the model placements in the scene. The placement of furniture fit the point clouds well.
• Figure 3: This illustration shows the design results from our automated pipeline. (Top row) The captured input image; (Middle row) The hybrid outline, which includes both the reconstructed gaussians and CAD models; (Bottom row) The virtual outline, where all elements are replaced by the CAD model.