LandMarkSystem Technical Report

TL;DR

This paper introduces LandMarkSystem, a novel computing framework designed to enhance multi-scale scene reconstruction and rendering by leveraging a componentized model adaptation layer and providing dedicated operators for complex 3D sparse computations, thus facilitating efficient training and rapid inference over extensive scenes.

Abstract

3D reconstruction is vital for applications in autonomous driving, virtual reality, augmented reality, and the metaverse. Recent advancements such as Neural Radiance Fields(NeRF) and 3D Gaussian Splatting (3DGS) have transformed the field, yet traditional deep learning frameworks struggle to meet the increasing demands for scene quality and scale. This paper introduces LandMarkSystem, a novel computing framework designed to enhance multi-scale scene reconstruction and rendering. By leveraging a componentized model adaptation layer, LandMarkSystem supports various NeRF and 3DGS structures while optimizing computational efficiency through distributed parallel computing and model parameter offloading. Our system addresses the limitations of existing frameworks, providing dedicated operators for complex 3D sparse computations, thus facilitating efficient training and rapid inference over extensive scenes. Key contributions include a modular architecture, a dynamic loading strategy for limited resources, and proven capabilities across multiple representative algorithms.This comprehensive solution aims to advance the efficiency and effectiveness of 3D reconstruction tasks.To facilitate further research and collaboration, the source code and documentation for the LandMarkSystem project are publicly available in an open-source repository, accessing the repository at: https://github.com/InternLandMark/LandMarkSystem.

Figures (14)

• Figure 1: LandMarkSystem supports various mainstream NeRF and 3DGS reconstruction algorithms. Through meticulous system optimization, it seamlessly adapts to scenes ranging from individual objects to entire cities and scales from single consumer-grade GPU to distributed clusters, delivering a smooth interactive experience on Web and VR platforms.
• Figure 2: LandMarkSystem Framework: The design of the LandMarkSystem is bifurcated into the Model Adaptation Layer and the Distributed Adaptation Layer. The Model Adaptation Layer equips the system with the capability to accommodate a variety of NeRF and 3DGS algorithms, ensuring compatibility and versatility. The Distributed Adaptation Layer, on the other hand, empowers the models with parallel processing capabilities. This tiered architecture not only facilitates the training and rendering of conventional small-scale object-level scenes but also seamlessly scales to address ultra-large-scale urban scenarios.
• Figure 3: LandMarkSystem Pipeline: The operational flow of the algorithm involves selecting between Rays or Camera inputs during the input stage, depending on the model. These inputs then enter the pipelines of the NeRF class and the 3DGaussian class, respectively, for training or rendering.
• Figure 4: LandMarkSystem Algorithm: Examples of modularizing algorithm components to build different algorithms
• Figure 5: Distributed API