TRIPS: Trilinear Point Splatting for Real-Time Radiance Field Rendering

TL;DR

TRilinear Point Splatting is presented, an approach that combines ideas from both Gaussian Splatting and ADOP and is entirely differentiable, allowing for automatic optimization of both point sizes and positions.

Abstract

Point-based radiance field rendering has demonstrated impressive results for novel view synthesis, offering a compelling blend of rendering quality and computational efficiency. However, also latest approaches in this domain are not without their shortcomings. 3D Gaussian Splatting [Kerbl and Kopanas et al. 2023] struggles when tasked with rendering highly detailed scenes, due to blurring and cloudy artifacts. On the other hand, ADOP [Rückert et al. 2022] can accommodate crisper images, but the neural reconstruction network decreases performance, it grapples with temporal instability and it is unable to effectively address large gaps in the point cloud. In this paper, we present TRIPS (Trilinear Point Splatting), an approach that combines ideas from both Gaussian Splatting and ADOP. The fundamental concept behind our novel technique involves rasterizing points into a screen-space image pyramid, with the selection of the pyramid layer determined by the projected point size. This approach allows rendering arbitrarily large points using a single trilinear write. A lightweight neural network is then used to reconstruct a hole-free image including detail beyond splat resolution. Importantly, our render pipeline is entirely differentiable, allowing for automatic optimization of both point sizes and positions. Our evaluation demonstrate that TRIPS surpasses existing state-of-the-art methods in terms of rendering quality while maintaining a real-time frame rate of 60 frames per second on readily available hardware. This performance extends to challenging scenarios, such as scenes featuring intricate geometry, expansive landscapes, and auto-exposed footage. The project page is located at: https://lfranke.github.io/trips/

Figures (10)

• Figure 1: Our pipeline: TRIPS renders and blends a point cloud trilinearly as 2$\times$2$\times$2 splats into multi-layered feature maps with the results being passed though our small neural network, containing only a single gated convolution per layer. Following, an optional spherical harmonics module and tone-mapper is used to produce the final image. This pipeline is completely differentiable, so that point descriptors (colors) and positions, as well as camera parameters are optimized via gradient descent.
• Figure 2: Trilinear Point Splatting: (left) all points and their respective size are projected into the target image. Based on this screen space size, each point is written to the correct layer of the image pyramid using a trilinear write (right). Large points are written to layers of lower resolution and therefore cover more space in the final image.
• Figure 3: In each pixel of the image pyramid, a depth-sorted list of colors and alpha values is stored. The final color of each pixel is computed using front-to-back alpha blending on the sorted list.
• Figure 4: Our design of one gated convolution block that processes the features of the image pyramid with the number of channels passed through indicated at each step.
• Figure 5: The initial COLMAP reconstruction lacks points on the pedestal of the statue (top left). Our approach distributes the few present points and increases their sizes (bottom left) thus rendering them also in lower layers (middle). Thus our pipeline can avoid distracting holes (right).