Clustered Error Correction with Grouped 4D Gaussian Splatting

TL;DR

The paper tackles dynamic novel-view synthesis with 4D Gaussian Splatting by addressing two core issues: ambiguous temporal correspondences and insufficient dynamic-region densification. It introduces Elliptical Error Clustering with Error Correcting Splat Addition to precisely localize and fix high-error regions, and Grouped 4D Gaussian Splatting to enforce shared motion transforms for stable, temporally coherent splat-object mappings. The approach yields state-of-the-art perceptual rendering quality and improved temporal stability on Neural 3D Video and Technicolor datasets, including notable PSNR gains (e.g., up to 0.39 dB on Technicolor) and reduced flicker across frames. The method is validated with extensive ablations and qualitative analyses, and the authors provide public release of implementation details and code.

Abstract

Existing 4D Gaussian Splatting (4DGS) methods struggle to accurately reconstruct dynamic scenes, often failing to resolve ambiguous pixel correspondences and inadequate densification in dynamic regions. We address these issues by introducing a novel method composed of two key components: (1) Elliptical Error Clustering and Error Correcting Splat Addition that pinpoints dynamic areas to improve and initialize fitting splats, and (2) Grouped 4D Gaussian Splatting that improves consistency of mapping between splats and represented dynamic objects. Specifically, we classify rendering errors into missing-color and occlusion types, then apply targeted corrections via backprojection or foreground splitting guided by cross-view color consistency. Evaluations on Neural 3D Video and Technicolor datasets demonstrate that our approach significantly improves temporal consistency and achieves state-of-the-art perceptual rendering quality, improving 0.39dB of PSNR on the Technicolor Light Field dataset. Our visualization shows improved alignment between splats and dynamic objects, and the error correction method's capability to identify errors and properly initialize new splats. Our implementation details and source code are available at https://github.com/tho-kn/cem-4dgs.

Figures (7)

• Figure 1: Failure of densification results in a lack of splat for the blue colored area on the green surface. By backprojecting the elliptical error cluster's ground truth color, we effectively add a Gaussian splat that corrects pixel error.
• Figure 3: Looking at the image as a whole, the correspondence between stripes is apparent between different frames. However, focusing on a small area like splats, the correspondence between them becomes ambiguous, as shown in the red box in the same coordinate. A splat to represent part of a stripe can be mapped to its neighboring stripe with a dynamic transform during the optimization process.
• Figure 4: Overall framework of our method. We aim to improve the rendering quality by introducing dynamic splats that correct observed pixel errors in the training set. Grouped 4DGS uses shared dynamic transforms for a group of dynamic splats to improve temporally consistent correspondence between splats and dynamic scene objects. (Sec. \ref{['sec:dynamic_group']}) Elliptical Error Clustering extracts error pixels and clusters pixels into an elliptical shape, ready for backprojection to initialize a Gaussian splat. (Sec. \ref{['sec:candidate_clustering']}) Error-Correcting Splat Addition corrects the pixel error of clusters caused by occlusion and lack of splats, by projecting the erroneous point's current depth to another view, comparing the ground-truth color, and applying foreground splitting or backproject addition. (Sec. \ref{['sec:splat_addition']}).
• Figure 5: Qualitative comparison of our method with state-of-the-art methods. From the top row, we show examples from the Sear Steak, Painter, and Train scenes from two datasets. The red and yellow box highlights the dynamic area with details, which is shown in a larger size below. For the Painter scene, brightness is adjusted from the ground-truth and rendering result for better visibility. The difference is pronounced in clearer teeth and a button, stripes in clothing, and sharper car window boundaries.
• Figure 6: Example of Foreground Splitting with our method in the Fabien scene. Hair has inconsistent details with the ground truth, occluding the background. Based on the error, ellipses are clustered for the correction candidate. Splats for hair are split (rendered with red color in the image).