LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scans

TL;DR

LiteReality is proposed, a novel pipeline that converts RGB-D scans of indoor environments into compact, realistic, and interactive 3D virtual replicas and introduces a training-free object retrieval module that achieves state-of-the-art similarity performance on the Scan2CAD benchmark, along with a robust material painting module capable of transferring appearances from images of any style to 3D assets.

Abstract

We propose LiteReality, a novel pipeline that converts RGB-D scans of indoor environments into compact, realistic, and interactive 3D virtual replicas. LiteReality not only reconstructs scenes that visually resemble reality but also supports key features essential for graphics pipelines -- such as object individuality, articulation, high-quality physically based rendering materials, and physically based interaction. At its core, LiteReality first performs scene understanding and parses the results into a coherent 3D layout and objects with the help of a structured scene graph. It then reconstructs the scene by retrieving the most visually similar 3D artist-crafted models from a curated asset database. Next, the Material Painting module enhances realism by recovering high-quality, spatially varying materials. Finally, the reconstructed scene is integrated into a simulation engine with basic physical properties to enable interactive behavior. The resulting scenes are compact, editable, and fully compatible with standard graphics pipelines, making them suitable for applications in AR/VR, gaming, robotics, and digital twins. In addition, LiteReality introduces a training-free object retrieval module that achieves state-of-the-art similarity performance on the Scan2CAD benchmark, along with a robust material painting module capable of transferring appearances from images of any style to 3D assets -- even under severe misalignment, occlusion, and poor lighting. We demonstrate the effectiveness of LiteReality on both real-life scans and public datasets. Project page: https://litereality.github.io; Video: https://www.youtube.com/watch?v=ecK9m3LXg2c

Figures (21)

• Figure 1: From RGB-D Scan to Graphics-Ready 3D Scene. LiteReality reconstructs compact, realistic 3D environments from real-world RGB-D scans, featuring near-photorealistic appearance, articulated geometry, and physically based rendering (PBR) materials—providing assets that can be easily integrated into simulation or rendering pipeline.
• Figure 2: Pipeline of LiteReality. Given input RGB-D scans, the process begins with scene perception and parsing, where room layouts and 3D object bounding boxes are detected and organized into a structured, physically plausible arrangement using a scene graph. In the object reconstruction stage, identical clustering first identifies repeated objects, followed by a hierarchical retrieval procedure that matches 3D models from the LiteReality database. The material painting stage retrieves and optimizes PBR materials by referencing the observed images. Finally, the procedural reconstruction stage assembles all components into a graphics-ready environment featuring realistic appearance and seamless integration with standard graphics pipelines.
• Figure 3: Procedural Reconstruction Stage. This stage progressively rebuilds rooms by reconstructing layouts, assembling dooors and windows, placing objects, and enabling interactive attributes.
• Figure 4: Materials Painting Stage Pipeline. Given a 3D model and reference images, this stage recovers realistic PBR materials that enhance visual realism for the 3D object.
• Figure 5: Graphics-Ready Reconstruction by LiteReality. Noisy indoor scans are converted into compact, realistic scenes with full PBR materials for all objects, ready for downstream graphics tasks.