MoE-GS: Mixture of Experts for Dynamic Gaussian Splatting

TL;DR

MoE-GS tackles the generalization gap in dynamic scene reconstruction by integrating multiple specialized Gaussian-based experts through a Volume-aware Pixel Router, enabling adaptive, spatially and temporally coherent blending. The method extends Mixture-of-Experts to dynamic Gaussian splatting, introduces per-Gaussian weights and differentiable weight splatting, and employs a two-stage training scheme to stabilize optimization. Efficiency is addressed via Single-Pass Multi-Expert Rendering and Gate-Aware Gaussian Pruning, while a knowledge-distillation pipeline allows individual experts to approximate full MoE performance without architectural changes. Experiments on N3V and Technicolor show state-of-the-art results across scenes, with distillation enhancing deployability for larger expert ensembles and a demonstrated transfer to monocular settings. Overall, MoE-GS offers a scalable, adaptive framework for robust dynamic scene reconstruction with practical deployment benefits.

Abstract

Recent advances in dynamic scene reconstruction have significantly benefited from 3D Gaussian Splatting, yet existing methods show inconsistent performance across diverse scenes, indicating no single approach effectively handles all dynamic challenges. To overcome these limitations, we propose Mixture of Experts for Dynamic Gaussian Splatting (MoE-GS), a unified framework integrating multiple specialized experts via a novel Volume-aware Pixel Router. Our router adaptively blends expert outputs by projecting volumetric Gaussian-level weights into pixel space through differentiable weight splatting, ensuring spatially and temporally coherent results. Although MoE-GS improves rendering quality, the increased model capacity and reduced FPS are inherent to the MoE architecture. To mitigate this, we explore two complementary directions: (1) single-pass multi-expert rendering and gate-aware Gaussian pruning, which improve efficiency within the MoE framework, and (2) a distillation strategy that transfers MoE performance to individual experts, enabling lightweight deployment without architectural changes. To the best of our knowledge, MoE-GS is the first approach incorporating Mixture-of-Experts techniques into dynamic Gaussian splatting. Extensive experiments on the N3V and Technicolor datasets demonstrate that MoE-GS consistently outperforms state-of-the-art methods with improved efficiency. Video demonstrations are available at https://anonymous.4open.science/w/MoE-GS-68BA/.

Figures (9)

• Figure 1: Overview of the MoE-GS framework. In Stage 1 (Expert Training), each expert is independently trained to reconstruct the dynamic scene by optimizing its own Gaussian representation, ensuring diverse modeling capabilities. In Stage 2 (Router Training), with all expert parameters fixed, the Volume-aware Pixel Router learns to dynamically blend expert-rendered images by computing spatially and temporally adaptive gating weights. The Candidate Experts (right) illustrate diverse Gaussian-based reconstruction methods integrated into our framework, including HexPlane Embedding-based, Per-Gaussian embedding-based, Interpolation-based, and Polynomial-based approaches, each suited for capturing different scene dynamics.
• Figure 2: Comparison of Router Architectures. The Pixel Router (top-left) assigns weights purely at the pixel level, ignoring volumetric features. The Volume Router (bottom-left) uses Gaussian-level weights but is difficult to optimize. Our Volume-aware Pixel Router (right) combines Gaussian-level weights with rasterization-based splatting.
• Figure 3: N3V Qualitative Results Comparison of our MoE-GS with other dynamic Gaussian splatting methods on Neural 3D Video dataset li2022neural. rgb]0.337, 0.671, 0.961Blue background highlight the method that produces the most visually accurate result among the baselines for each region.
• Figure 4: Qualitative comparison of MoE Router variants. Each router applies a different routing strategy, resulting in varied structural consistency and detail across the outputs.
• Figure 5: Architectural details of the Volume-aware Pixel Router.