GPS-Gaussian+: Generalizable Pixel-wise 3D Gaussian Splatting for Real-Time Human-Scene Rendering from Sparse Views

TL;DR

This work proposes a generalizable Gaussian Splatting approach for high-resolution image rendering under a sparse-view camera setting and introduces Gaussian parameter maps defined on the source views and directly regress Gaussian properties for instant novel view synthesis without any fine-tuning or optimization.

Abstract

Differentiable rendering techniques have recently shown promising results for free-viewpoint video synthesis of characters. However, such methods, either Gaussian Splatting or neural implicit rendering, typically necessitate per-subject optimization which does not meet the requirement of real-time rendering in an interactive application. We propose a generalizable Gaussian Splatting approach for high-resolution image rendering under a sparse-view camera setting. To this end, we introduce Gaussian parameter maps defined on the source views and directly regress Gaussian properties for instant novel view synthesis without any fine-tuning or optimization. We train our Gaussian parameter regression module on human-only data or human-scene data, jointly with a depth estimation module to lift 2D parameter maps to 3D space. The proposed framework is fully differentiable with both depth and rendering supervision or with only rendering supervision. We further introduce a regularization term and an epipolar attention mechanism to preserve geometry consistency between two source views, especially when neglecting depth supervision. Experiments on several datasets demonstrate that our method outperforms state-of-the-art methods while achieving an exceeding rendering speed.

Figures (13)

• Figure 1: High-fidelity and real-time rendering. On the top, GPS-Gaussian produces $2K$-resolution rendering of character, while GPS-Gaussian+ renders novel views of human-centered scenes on the bottom. Our methods outperform the state-of-the-art feed-forward implicit rendering method ENeRF lin2022enerf, explicit rendering method MVSplat chen2024mvsplat and optimization-based methods 3D-GS kerbl2023_3dgs and 4D-GS Wu2024_4dgs.
• Figure 2: Overview of GPS-Gaussian+. Given RGB images of a human-centered scene with sparse camera views and a target novel viewpoint, we select the adjacent two views on which to formulate our pixel-wise Gaussian representation. We extract the image features by using epipolar attention and then conduct an iterative depth estimation. For each source view, the RGB image serves as a color map, while the other parameters of 3D Gaussians are predicted in a pixel-wise manner. The Gaussian parameter maps defined on 2D image planes of both views are further unprojected to 3D space via refined depth maps and aggregated for novel view rendering. The fully differentiable framework enables a joint training mechanism with only rendering loss and geometry regularization.
• Figure 3: Qualitative comparison on human-scene data. Our method produces high-quality renderings with respect to others.
• Figure 4: Qualitative comparison on human-only data. Our method produces more detailed human appearances and can recover more reasonable geometry.
• Figure 5: Qualitative ablation study on GPS-Gaussian/GPS-Gaussian+ with different supervision settings. We show the effectiveness of the integration in GPS-Gaussian+ when neglecting depth supervision.