NeuroGauss4D-PCI: 4D Neural Fields and Gaussian Deformation Fields for Point Cloud Interpolation

TL;DR

NeuroGauss4D-PCI tackles point cloud frame interpolation by modeling 4D spatio-temporal dynamics with a hybrid of Gaussian representations and a latent 4D neural field. It introduces an iterative Gaussian cloud soft clustering scheme, a Temporal Radial Basis Function Gaussian Residual to interpolate Gaussian parameters over time, and a 4D Gaussian deformation field learned via temporal graph convolutions, all fused through an efficient latent-geometric attention mechanism to predict intermediate frames. The approach yields state-of-the-art performance on DHB and NL-Drive PCI benchmarks and strong results on 3D scene flow KITTI datasets, while maintaining a compact parameter count and supporting potential auto-labeling and densification tasks. The work advances temporal point cloud understanding by coupling probabilistic Gaussian structures with continuous 4D neural representations, enabling accurate, smooth, and scalable dynamic scene reconstruction in LiDAR-equipped systems.

Abstract

Point Cloud Interpolation confronts challenges from point sparsity, complex spatiotemporal dynamics, and the difficulty of deriving complete 3D point clouds from sparse temporal information. This paper presents NeuroGauss4D-PCI, which excels at modeling complex non-rigid deformations across varied dynamic scenes. The method begins with an iterative Gaussian cloud soft clustering module, offering structured temporal point cloud representations. The proposed temporal radial basis function Gaussian residual utilizes Gaussian parameter interpolation over time, enabling smooth parameter transitions and capturing temporal residuals of Gaussian distributions. Additionally, a 4D Gaussian deformation field tracks the evolution of these parameters, creating continuous spatiotemporal deformation fields. A 4D neural field transforms low-dimensional spatiotemporal coordinates ($x,y,z,t$) into a high-dimensional latent space. Finally, we adaptively and efficiently fuse the latent features from neural fields and the geometric features from Gaussian deformation fields. NeuroGauss4D-PCI outperforms existing methods in point cloud frame interpolation, delivering leading performance on both object-level (DHB) and large-scale autonomous driving datasets (NL-Drive), with scalability to auto-labeling and point cloud densification tasks. The source code is released at https://github.com/jiangchaokang/NeuroGauss4D-PCI.

Figures (7)

• Figure 1: NeuroGauss4D-PCI robustly outperforms existing methods zheng2023neuralpcijiang20243dsflabelling across multiple datasets, frame intervals, and point cloud densities, consistently achieving lower interpolation errors. NeuroGauss4D-PCI robustly handles minute non-rigid deformations, large-scale unstructured scenes, dynamic environments with non-uniform data, and extensive motions. The proposed method consistently achieves precise local and global point cloud predictions.
• Figure 2: Three key steps of NeuroGauss4D-PCI: 1) Latent feature learning and point cloud Gaussian representation: Fourier feature mappings and 4D neural fields map low-dimensional temporal coordinates ($x, y, z, t$) to high-dimensional latent features, while representing the original temporal point clouds as robust multi-Gaussian spheres ($\mu, \Sigma, \Phi$). 2) The temporal radial basis function Gaussian residual (RBF-GR) module captures residuals among temporal Gaussian distributions, fusing smooth temporal Gaussian distributions with latent features to construct a 4D Gaussian deformation field that learns and smoothens point cloud deformations. 3) An efficient transformer architecture aggregates features from the 4D deformation field and latent features, enabling point cloud interpolations at any given timestamp through a point cloud prediction head.
• Figure 3: Temporal radial basis function Gaussian residual (RBF-GR) Module. The normalized RBF weights $\tilde{\zeta}_i^{rbf}(t_2)$ are used to compute the residuals of Gaussian means $\Delta\mu_{t_1\rightarrow t_2}^{(\theta)}$, rotations $\Delta R_{t_1\rightarrow t_2}^{(\theta)}$, and features $\Delta \Phi^{feat}_{t_1\rightarrow t_2}$ between time $t_1$ and $t_2$. The covariance matrix $\Sigma_{t_2}$ is then updated using the learned rotation residual. $W_{rbf} = \sigma_{softmax}(MLP(\Phi^{feat}_{t1}))$, where $W_{rbf}$ denotes the attention weights for RBF activations, adaptively adjusted based on Gaussian features.
• Figure 4: 4D Gaussian Deformation Field. Utilizing Gaussian means $\mu$, covariances $\Sigma$, and features $\Phi^{feat}$, and time ($t_1,t_2$, $\cdots$) as inputs, the spatio-temporal graph convolutional network captures spatio-temporal patterns to learn continuous 4D deformation fields. The Gaussian Representation pooling module projects the point cloud onto the Gaussian sphere and upsamples, using max-pooling to extract salient features while capturing complex point dynamics and temporal evolution.
• Figure 5: Visualization of NeuroGauss4D-PCI for multi-sensor time synchronization and point cloud densification applications. In point cloud densification, sparse ground truth points are shown in green, while the predicted dense point cloud is shown in red, exhibiting good overlap with the sparse ground truth.