Constrained Dynamic Gaussian Splatting

TL;DR

CDGS tackles memory bottlenecks in dynamic 4D scene reconstruction by enforcing a hard Gaussian budget during training. It introduces a differentiable budget controller driven by a unified importance score that blends geometric stability, motion significance, and perceptual impact, enabling precise densification and pruning under a fixed budget $N_{\text{target}}$. An adaptive static-dynamic allocation automatically partitions Gaussians into static and dynamic sets, while a three-phase training regime and a dual-mode compression pipeline ensure budget adherence and storage efficiency. Across multiple datasets, CDGS achieves superior rate-distortion performance and up to a 3x reduction in model size, enabling efficient edge-device deployment.

Abstract

While Dynamic Gaussian Splatting enables high-fidelity 4D reconstruction, its deployment is severely hindered by a fundamental dilemma: unconstrained densification leads to excessive memory consumption incompatible with edge devices, whereas heuristic pruning fails to achieve optimal rendering quality under preset Gaussian budgets. In this work, we propose Constrained Dynamic Gaussian Splatting (CDGS), a novel framework that formulates dynamic scene reconstruction as a budget-constrained optimization problem to enforce a strict, user-defined Gaussian budget during training. Our key insight is to introduce a differentiable budget controller as the core optimization driver. Guided by a multi-modal unified importance score, this controller fuses geometric, motion, and perceptual cues for precise capacity regulation. To maximize the utility of this fixed budget, we further decouple the optimization of static and dynamic elements, employing an adaptive allocation mechanism that dynamically distributes capacity based on motion complexity. Furthermore, we implement a three-phase training strategy to seamlessly integrate these constraints, ensuring precise adherence to the target count. Coupled with a dual-mode hybrid compression scheme, CDGS not only strictly adheres to hardware constraints (error < 2%}) but also pushes the Pareto frontier of rate-distortion performance. Extensive experiments demonstrate that CDGS delivers optimal rendering quality under varying capacity limits, achieving over 3x compression compared to state-of-the-art methods.

Figures (10)

• Figure 1: Left: Our CDGS leverages differentiable budget control for precise Gaussian number regulation, achieving adaptive static-dynamic allocation across varying target numbers and optimal rendering quality. Middle: Visual comparison with state-of-the-art methods, highlighting advantages in visual quality, model size, and Gaussian count controllability (second row: actual/target counts). Right: Superior rate-distortion performance and precise Gaussian number control of our approach, outperforming all prior works (e.g. 4DGS yang2023gs4d, STGS Li_STG_2024_CVPR, Ex4DGS lee2024ex4dgs).
• Figure 2: Overview of the proposed CDGS framework. (Top) Three-Phase Pipeline: The training progresses from a Warm-up phase to establish foundational priors, through a Budget Enforcement phase where constraints are actively applied, to a final Fine-tuning phase that maximizes quality under the fixed count $N_{\text{target}}$. (Bottom Left) Adaptive Dynamic-Static Allocation: This module analyzes the distribution of motion magnitudes to identify a natural separation threshold $\tau_{\text{motion}}$ via peak identification. It decomposes the scene into a Static set $\mathcal{S}$ and a Dynamic set $\mathcal{D}(t)$ . (Bottom Right) Differentiable Budget Control: This module regulates capacity by computing a Unified Importance Score $M_i$. It fuses Perceptual cues $\mathcal{F}_{\text{perceptual}}$ and Geometric/Motion cues $\mathcal{F}_{\text{geom/motion}}$. The resulting score guides the differentiable budget loss $\mathcal{L}_{\text{budget}}$ to precisely prune or densify Gaussians towards $N_{\text{target}}$.
• Figure 3: Illustration of the Differentiable Budget Controller. The controller aggregates geometric, motion, and perceptual cues to compute a Unified Importance Score $M_i$. This score passes through a hard-sigmoid gate to estimate the effective Gaussian count $N_p$, which is strictly regulated towards the target $N_{\text{target}}$ via a quadratic budget loss $\mathcal{L}_{\text{budget}}$. Guided by this budget constraint, the closed-loop policy performs densification on high-importance Gaussians and pruning on low-importance ones to dynamically optimize capacity.
• Figure 4: Illustration of our dual-mode hybrid compression strategy, which separates outliers from both static and dynamic Gaussian data, then applies distinct compression approaches.
• Figure 5: Rate-distortion curves on different datasets, illustrating the superiority of our method over ReRF rerf, TeTriRF tetrirf, 4DGC hu20254dgc, 4DGCPro zheng20254dgcpro, RD4DGS lee2025rd4dgs and Ex4DGS lee2024ex4dgs.