MAC-Ego3D: Multi-Agent Gaussian Consensus for Real-Time Collaborative Ego-Motion and Photorealistic 3D Reconstruction

TL;DR

MAC-Ego3D addresses real-time, multi-agent ego-motion and photorealistic 3D reconstruction by introducing a unified Gaussian splat map and a two-level consensus mechanism. The framework enables parallel intra-agent mapping and asynchronous inter-agent alignment, achieving dense, coherent maps with high fidelity through differentiable rendering. Empirical results on synthetic Replica and real-world 7-Scenes datasets show substantial gains in trajectory accuracy, RGB-D rendering quality, and runtime efficiency, with robustness to noise. This approach advances scalable spatial intelligence by combining continuous Gaussian representations with scalable, parallel collaboration across agents.

Abstract

Real-time multi-agent collaboration for ego-motion estimation and high-fidelity 3D reconstruction is vital for scalable spatial intelligence. However, traditional methods produce sparse, low-detail maps, while recent dense mapping approaches struggle with high latency. To overcome these challenges, we present MAC-Ego3D, a novel framework for real-time collaborative photorealistic 3D reconstruction via Multi-Agent Gaussian Consensus. MAC-Ego3D enables agents to independently construct, align, and iteratively refine local maps using a unified Gaussian splat representation. Through Intra-Agent Gaussian Consensus, it enforces spatial coherence among neighboring Gaussian splats within an agent. For global alignment, parallelized Inter-Agent Gaussian Consensus, which asynchronously aligns and optimizes local maps by regularizing multi-agent Gaussian splats, seamlessly integrates them into a high-fidelity 3D model. Leveraging Gaussian primitives, MAC-Ego3D supports efficient RGB-D rendering, enabling rapid inter-agent Gaussian association and alignment. MAC-Ego3D bridges local precision and global coherence, delivering higher efficiency, largely reducing localization error, and improving mapping fidelity. It establishes a new SOTA on synthetic and real-world benchmarks, achieving a 15x increase in inference speed, order-of-magnitude reductions in ego-motion estimation error for partial cases, and RGB PSNR gains of 4 to 10 dB. Our code will be made publicly available at https://github.com/Xiaohao-Xu/MAC-Ego3D .

Figures (23)

• Figure 1: Towards collaborative, real-time, photorealistic 3D reconstruction in multi-agent systems. (a) In MAC-Ego3D, each agent independently captures observations, estimates ego-motion, and constructs a local Gaussian-based 3D map, which is then periodically aligned with others for collaborative optimization. (b) Multi-Agent Gaussian Consensus, with intra- and inter-agent Gaussian selection, association, alignment, and optimization, enabling efficient tracking, rapid loop closure and high-fidelity mapping.
• Figure 2: Pipeline overview of MAC-Ego3D. MAC-Ego3D leverages parallel Intra-Agent Gaussian Consensus and periodic Inter-Agent Gaussian Consensus to enable real-time pose tracking and photorealistic 3D reconstruction using a shared 3D Gaussian map representation.
• Figure 3: Qualitative RGB image rendering quality comparison between multi-agent SLAM models with dense reconstruction capability, i.e., CP-SLAM and our MAC-Ego3D, on Multi-agent Replica (Left) and 7-Scenes (Right) datasets.
• Figure 4: Qualitative depth rendering comparison on 7-Scenes.
• Figure 5: Multi-agent trajectory estimation results on 7-Scenes.