Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-robot autonomous 3D reconstruction using Gaussian splatting with Semantic guidance

Jing Zeng, Qi Ye, Tianle Liu, Yang Xu, Jin Li, Jinming Xu, Liang Li, Jiming Chen

TL;DR

This work tackles scalable indoor 3D reconstruction by introducing a centralized multi-robot framework based on 3D Gaussian splatting (3DGS). It integrates online open-vocabulary semantic segmentation to bias viewpoint sampling toward high-uncertainty surface regions, and couples this with a hierarchical planning strategy that separates mode assignment and task clustering to expedite MDMTSP solutions. Key contributions include the first centralized multi-robot 3DGS pipeline, semantic-guided reconstruction task generation, and an efficient global-local collaboration scheme that improves reconstruction quality and planning efficiency. The approach yields superior performance over existing planning methods in simulations and real-robot experiments, enabling faster, higher-fidelity scene completion without requiring heavy model databases or point-cloud completions.

Abstract

Implicit neural representations and 3D Gaussian splatting (3DGS) have shown great potential for scene reconstruction. Recent studies have expanded their applications in autonomous reconstruction through task assignment methods. However, these methods are mainly limited to single robot, and rapid reconstruction of large-scale scenes remains challenging. Additionally, task-driven planning based on surface uncertainty is prone to being trapped in local optima. To this end, we propose the first 3DGS-based centralized multi-robot autonomous 3D reconstruction framework. To further reduce time cost of task generation and improve reconstruction quality, we integrate online open-vocabulary semantic segmentation with surface uncertainty of 3DGS, focusing view sampling on regions with high instance uncertainty. Finally, we develop a multi-robot collaboration strategy with mode and task assignments improving reconstruction quality while ensuring planning efficiency. Our method demonstrates the highest reconstruction quality among all planning methods and superior planning efficiency compared to existing multi-robot methods. We deploy our method on multiple robots, and results show that it can effectively plan view paths and reconstruct scenes with high quality.

Multi-robot autonomous 3D reconstruction using Gaussian splatting with Semantic guidance

TL;DR

This work tackles scalable indoor 3D reconstruction by introducing a centralized multi-robot framework based on 3D Gaussian splatting (3DGS). It integrates online open-vocabulary semantic segmentation to bias viewpoint sampling toward high-uncertainty surface regions, and couples this with a hierarchical planning strategy that separates mode assignment and task clustering to expedite MDMTSP solutions. Key contributions include the first centralized multi-robot 3DGS pipeline, semantic-guided reconstruction task generation, and an efficient global-local collaboration scheme that improves reconstruction quality and planning efficiency. The approach yields superior performance over existing planning methods in simulations and real-robot experiments, enabling faster, higher-fidelity scene completion without requiring heavy model databases or point-cloud completions.

Abstract

Implicit neural representations and 3D Gaussian splatting (3DGS) have shown great potential for scene reconstruction. Recent studies have expanded their applications in autonomous reconstruction through task assignment methods. However, these methods are mainly limited to single robot, and rapid reconstruction of large-scale scenes remains challenging. Additionally, task-driven planning based on surface uncertainty is prone to being trapped in local optima. To this end, we propose the first 3DGS-based centralized multi-robot autonomous 3D reconstruction framework. To further reduce time cost of task generation and improve reconstruction quality, we integrate online open-vocabulary semantic segmentation with surface uncertainty of 3DGS, focusing view sampling on regions with high instance uncertainty. Finally, we develop a multi-robot collaboration strategy with mode and task assignments improving reconstruction quality while ensuring planning efficiency. Our method demonstrates the highest reconstruction quality among all planning methods and superior planning efficiency compared to existing multi-robot methods. We deploy our method on multiple robots, and results show that it can effectively plan view paths and reconstruct scenes with high quality.

Paper Structure

This paper contains 28 sections, 8 equations, 4 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Autonomous 3D reconstruction
  4. Multiple-robot collaboration
  5. Semantic guidance in planning
  6. Method
  7. Problem Statement and System Overview
  8. Perception
  9. 3D Gaussian splatting with surface uncertainty
  10. ASH for scene geometry and pruning noisy Gaussians
  11. Open-vocabulary online 3D semantic segmentation
  12. Task generation
  13. Exploration task generation
  14. Semantic-guided reconstruction task generation
  15. Hierarchical planning
  16. ...and 13 more sections

Figures (4)

  • Figure 1: The pipeline of our proposed method.
  • Figure 2: Overview of our method. (a) Semantic-guided reconstruction task generation. (b) Multi-robot collaboration for mode and task assignments. (c) Single-robot view path planning.
  • Figure 3: Comparison with different methods. Top: novel view synthesis from 3DGS; Bottom: reconstructed meshes from ASH.
  • Figure 4: Comparison of trajectories with different methods.