Radiative Gaussian Splatting for Efficient X-ray Novel View Synthesis

TL;DR

A 3D Gaussian splatting-based framework, namely X-Gaussian, for X-ray novel view synthesis that outperforms state-of-the-art methods by 6.5 dB while enjoying less than 15% training time and over 73x inference speed.

Abstract

X-ray is widely applied for transmission imaging due to its stronger penetration than natural light. When rendering novel view X-ray projections, existing methods mainly based on NeRF suffer from long training time and slow inference speed. In this paper, we propose a 3D Gaussian splatting-based framework, namely X-Gaussian, for X-ray novel view synthesis. Firstly, we redesign a radiative Gaussian point cloud model inspired by the isotropic nature of X-ray imaging. Our model excludes the influence of view direction when learning to predict the radiation intensity of 3D points. Based on this model, we develop a Differentiable Radiative Rasterization (DRR) with CUDA implementation. Secondly, we customize an Angle-pose Cuboid Uniform Initialization (ACUI) strategy that directly uses the parameters of the X-ray scanner to compute the camera information and then uniformly samples point positions within a cuboid enclosing the scanned object. Experiments show that our X-Gaussian outperforms state-of-the-art methods by 6.5 dB while enjoying less than 15% training time and over 73x inference speed. The application on sparse-view CT reconstruction also reveals the practical values of our method. Code is publicly available at https://github.com/caiyuanhao1998/X-Gaussian . A video demo of the training process visualization is at https://www.youtube.com/watch?v=gDVf_Ngeghg .

Figures (10)

• Figure 1: PSNR-minute-fps comparison. The radius of circle represents the training time (minutes). Our method is the most efficient.
• Figure 2: Point cloud visualization of the original 3DGS 3dgs (top) and our X-Gaussian (bottom). We visualize the positions and opacities of the Gaussian point clouds at different training iterations. We also visualize the volume of foot as a reference. Note that the volume is not the ground truth of point clouds. Our X-Gaussian can better represent the detailed structures than 3DGS, showing faster and better convergence.
• Figure 3: Pipeline of our method. (a) Angle-pose Cuboid Uniform Initialization (ACUI) strategy uses the parameters of X-ray scanner to compute intrinsic and extrinsic matrices, and samples center points for 3D Gaussians. (b) Our radiative Gaussian point cloud model learns to predict the radiation intensity of 3D points. (c) Based on our Gaussian model, we develop a GPU-friendly Differentiable Radiative Rasterization (DRR).
• Figure 4: Comparison between the Gaussian point cloud models of the original 3DGS and our X-Gaussian. (a) The original RGB Gaussian point cloud model uses spherical harmonics (SH) to simulate the anisotropic natural light distribution and view-dependent color. (b) Our radiative Gaussian point cloud model employs the weighted sum of point features to fit the isotropic X-ray penetration and view-independent radiation intensity.
• Figure 5: Qualitative results of novel view synthesis on the scenes of pancreas (top) and chest (bottom). Our X-Gaussian yields clearer results. Please zoom in for a better view.