SimRecon: SimReady Compositional Scene Reconstruction from Real Videos

TL;DR

SimRecon is proposed, a framework that realizes a "Perception-Generation-Simulation" pipeline towards cluttered scene reconstruction, which first conducts scene-level semantic reconstruction from video input, then performs single-object generation, and finally assembles these assets in the simulator.

Abstract

Compositional scene reconstruction seeks to create object-centric representations rather than holistic scenes from real-world videos, which is natively applicable for simulation and interaction. Conventional compositional reconstruction approaches primarily emphasize on visual appearance and show limited generalization ability to real-world scenarios. In this paper, we propose SimRecon, a framework that realizes a "Perception-Generation-Simulation" pipeline towards cluttered scene reconstruction, which first conducts scene-level semantic reconstruction from video input, then performs single-object generation, and finally assembles these assets in the simulator. However, naively combining these three stages leads to visual infidelity of generated assets and physical implausibility of the final scene, a problem particularly severe for complex scenes. Thus, we further propose two bridging modules between the three stages to address this problem. To be specific, for the transition from Perception to Generation, critical for visual fidelity, we introduce Active Viewpoint Optimization, which actively searches in 3D space to acquire optimal projected images as conditions for single-object completion. Moreover, for the transition from Generation to Simulation, essential for physical plausibility, we propose a Scene Graph Synthesizer, which guides the construction from scratch in 3D simulators, mirroring the native, constructive principle of the real world. Extensive experiments on the ScanNet dataset validate our method's superior performance over previous state-of-the-art approaches.

Figures (8)

• Figure 1: We propose SimRecon, a framework for reconstructing simulation-ready, compositional 3D scenes from real-world videos. Our method introduces a "Perception-Generation-Simulation" pipeline which transforms cluttered input videos into physically assembled scenes. To ensure visually faithful generated assets and physically plausible final scenes, we propose two bridging strategies: Active View Optimization to acquire optimal generation conditions, and Constructive Assembly to follow a native building principle.
• Figure 2: The overall framework of our approach SimRecon. We propose a "Perception-Generation-Simulation" pipeline with object-centric scene representations towards compositional 3D scene reconstruction from cluttered video input. In this figure, we provide illustrative visualizations using the backpack as the example to introduce our two core modules: Active Viewpoint Optimization (AVO) and Scene Graph Synthesizer (SGS). There, we visualize a semantic-level graph for clarity, while our framework operates at the instance-level.
• Figure 3: Qualitative Comparison for Compositional 3D Reconstruction. We present qualitative visualizations of the final reconstructed scenes. For single-view setting, we render the 3D representation at the target viewpoint as the input for these methods.
• Figure 4: Qualitative comparison of viewpoint sampling strategies. We uniformly use a single image as the condition and utilize the same generative model.
• Figure 5: Qualitative comparison of physical scene construction in the simulator.