Fed3DGS: Scalable 3D Gaussian Splatting with Federated Learning

TL;DR

Fed3DGS tackles city-scale 3D reconstruction by marrying 3D Gaussian Splatting with federated learning, enabling decentralized scene modeling with scalable global updates. It introduces a distillation-based model merge strategy and an appearance encoding mechanism to handle non-IID data and seasonal appearance changes, achieving rendering quality competitive with centralized baselines while reducing global model size and per-client training time. Experiments across Mill 19, UrbanScene3D, Quad 6k, and 4Seasons demonstrate strong SSIM and LPIPS performance and show the method’s ability to reflect continuous appearance changes. The work highlights practical advantages for scalable, maintainable 3D reconstruction and points to future directions in global pose handling and background modeling.

Abstract

In this work, we present Fed3DGS, a scalable 3D reconstruction framework based on 3D Gaussian splatting (3DGS) with federated learning. Existing city-scale reconstruction methods typically adopt a centralized approach, which gathers all data in a central server and reconstructs scenes. The approach hampers scalability because it places a heavy load on the server and demands extensive data storage when reconstructing scenes on a scale beyond city-scale. In pursuit of a more scalable 3D reconstruction, we propose a federated learning framework with 3DGS, which is a decentralized framework and can potentially use distributed computational resources across millions of clients. We tailor a distillation-based model update scheme for 3DGS and introduce appearance modeling for handling non-IID data in the scenario of 3D reconstruction with federated learning. We simulate our method on several large-scale benchmarks, and our method demonstrates rendered image quality comparable to centralized approaches. In addition, we also simulate our method with data collected in different seasons, demonstrating that our framework can reflect changes in the scenes and our appearance modeling captures changes due to seasonal variations.

Figures (14)

• Figure 1: The proposed federated learning framework. Clients collaboratively train a global model under the orchestration of a central server. Our framework continuously updates the global model by repeating steps 1 to 3.
• Figure 2: Illustration of the distillation-based model update. The result of distillation only with local cameras (i.e., cameras used to train a local model) has noisy Gaussians.
• Figure 3: Rendered RGB images and depth images from test views. Fed3DGS captures the detailed structures (e.g., the steel frames in the top row and the distant buildings in the bottom row).
• Figure 4: The number of Gaussians and PSNR with and without reset opacity (reset op) and entropy minimization (entropy min.) on the building scene. We show the mean and standard deviation over five trials. Baseline indicates a result without both reset opacity and entropy minimization.
• Figure 5: The histogram of the number of Gaussians.