Grid4D: 4D Decomposed Hash Encoding for High-Fidelity Dynamic Gaussian Splatting

TL;DR

Grid4D addresses the challenge of high-fidelity dynamic scene rendering by replacing low-rank plane-based 4D encodings with a tri-axial, 4D-decomposed hash encoding that splits $(x,y,z,t)$ into four 3D hashes. A directional attention mechanism fuses spatial and temporal features to predict precise Gaussian deformations, while a smooth regularization term stabilizes training and reduces deformation chaos. The approach yields state-of-the-art visual quality and rendering speed on multiple dynamic-scene benchmarks, outperforming plane-based and some implicit methods, with scalable rendering for large Gaussian sets. Limitations include no improvement in training speed and potential artifacts in scenes with very large or complex motions, pointing to avenues for future enhancements in stability and efficiency.

Abstract

Recently, Gaussian splatting has received more and more attention in the field of static scene rendering. Due to the low computational overhead and inherent flexibility of explicit representations, plane-based explicit methods are popular ways to predict deformations for Gaussian-based dynamic scene rendering models. However, plane-based methods rely on the inappropriate low-rank assumption and excessively decompose the space-time 4D encoding, resulting in overmuch feature overlap and unsatisfactory rendering quality. To tackle these problems, we propose Grid4D, a dynamic scene rendering model based on Gaussian splatting and employing a novel explicit encoding method for the 4D input through the hash encoding. Different from plane-based explicit representations, we decompose the 4D encoding into one spatial and three temporal 3D hash encodings without the low-rank assumption. Additionally, we design a novel attention module that generates the attention scores in a directional range to aggregate the spatial and temporal features. The directional attention enables Grid4D to more accurately fit the diverse deformations across distinct scene components based on the spatial encoded features. Moreover, to mitigate the inherent lack of smoothness in explicit representation methods, we introduce a smooth regularization term that keeps our model from the chaos of deformation prediction. Our experiments demonstrate that Grid4D significantly outperforms the state-of-the-art models in visual quality and rendering speed.

Figures (13)

• Figure 1: We propose a novel explicit representation method for dynamic scene rendering that decomposes the space-time 4D encoding into the 3D format without the unsuitable low-rank assumption. We achieve significant improvements over the state-of-the-art models 4d-gsdeformable-3d-gaussians in rendering quality.
• Figure 2: Comparison of our proposed 4D decomposed hash encoding with the plane-based explicit representation 4d-gs. (a) Compared to the plane-based methods based on the low-rank assumption, our methods reduce the overlap ratio in the features from a half to a quarter when encoding points A and B with heavily overlapping coordinates. (b) is the t-SNE t-sne visualization of all the features, and the colors denote the corresponding represented deformations. The diversity of colors demonstrates that the reduced overlap makes the features represent different deformations more effectively.
• Figure 3: The overview of Grid4D. Given the canonical Gaussians and the timestamp, we first encode the decomposed input separately. Then we apply the directional attention scores generated by the spatial static features to the temporal dynamic features, and we decode the features with a tiny multi-head MLP. Finally, the Gaussians deformed by the predicted deformations are splatted by the differentiable rasterization operation gaussian-splatting to render the images for supervision.
• Figure 4: Qualitative comparisons on the synthetic D-NeRF dataset d-nerf with our baselines tineuvox4d-gsdeformable-3d-gaussians.
• Figure 5: Qualitative comparisons on the real-world HyperNeRF hypernerf dataset.