MoE3D: A Mixture-of-Experts Module for 3D Reconstruction

TL;DR

MoE3D introduces a lightweight mixture-of-experts head to depth prediction in feed-forward 3D reconstruction, enabling per-pixel routing among multiple depth hypotheses to better handle boundary uncertainty. By integrating this head with a pre-trained VGGT backbone and training with entropy regularization, the approach sharpens depth boundaries and reduces flying-point artifacts while maintaining modest computational overhead. The method yields state-of-the-art or competitive performance across monocular and multi-view reconstruction benchmarks, including substantial gains on boundary-focused metrics and indoor scenes. This plug-in design offers a practical path toward more accurate and perceptually faithful 3D reconstructions without heavy training or inference costs.

Abstract

MoE3D is a mixture-of-experts module designed to sharpen depth boundaries and mitigate flying-point artifacts (highlighted in red) of existing feed-forward 3D reconstruction models (left side). MoE3D predicts multiple candidate depth maps and fuses them via dynamic weighting (visualized by MoE weights on the right side). When integrated with a pre-trained 3D reconstruction backbone such as VGGT, it substantially enhances reconstruction quality with minimal additional computational overhead. Best viewed digitally.

Figures (14)

• Figure 1: MoE3D is a mixture-of-experts module designed to sharpen depth boundaries and mitigate flying-point artifacts (highlighted in red) of existing feed-forward 3D reconstruction models (left side). MoE3D predicts multiple candidate depth maps and fuses them via dynamic weighting (visualized by MoE weights on the right side). When integrated with a pre-trained 3D reconstruction backbone such as VGGT, it substantially enhances reconstruction quality with minimal additional computational overhead. Best viewed digitally.
• Figure 2: Architecture Overview. We extend the VGGT backbone with a Mixture-of-Experts (MoE) head for depth estimation. The MoE head replaces the DPT head with $K$ expert branches and a gating network that dynamically routes features across experts, improving boundary sharpness and reducing flying-point artifacts.
• Figure 3: Effect of Entropy Regularization. Visualization of gating assignments (argmax) for four experts (red, blue, green, yellow). Without entropy regularization, the experts exhibit weak specialization. Large regularization values ($\lambda \!\geq\! 10^{-3}$) cause premature collapse to one or two experts, whereas smaller values yield sharper spatial partitions and lower final loss. At $\lambda = 10^{-4}$, the experts specialize distinctly, each capturing different orientations of depth boundaries.
• Figure 4: Qualitative results of multi-view 3D reconstruction. Each group shows input views (top) and reconstructed point clouds by VGGT (middle) and our MoE3D (bottom). Red boxes highlight regions where VGGT exhibits blurred geometry or flying points.
• Figure 5: Qualitative results on monocular depth estimation. From top to bottom: Bonn, a stylized anime image, and KITTI. The right column shows point-cloud reconstructions from predicted depths. MoE3D produces sharper boundaries and significantly reduces flying-point artifacts compared to VGGT across diverse domains. Zoom in to view details.