From Volume Rendering to 3D Gaussian Splatting: Theory and Applications

TL;DR

The paper surveys 3D Gaussian Splatting (3DGS) as a transformative approach for 3D reconstruction from posed images, leveraging explicit 3D Gaussians to enable real-time rendering via volume splatting and avoiding the costly empty-space queries of neural radiance fields. It derives 3DGS from the volume rendering equation, details Gaussian initialization from SfM, and describes adaptive training strategies that modify the Gaussian set to maintain coverage and detail. The work further catalogs extensions addressing memory efficiency, aliasing, and lighting, and surveys broad applications including surface reconstruction, avatars, animation, and feed-forward reconstruction from sparse views, with diffusion-based content generation as a rising trend. Overall, the paper positions 3DGS as a scalable, pipeline-friendly alternative to NeRFs with strong real-time rendering capabilities and expanding capabilities for dynamic content and downstream tasks. The findings highlight practical impact in fast 3D reconstruction workflows, high-fidelity view synthesis, and versatile 3D content generation, while outlining clear open problems in Gaussian optimization, render accuracy, and relighting.

Abstract

The problem of 3D reconstruction from posed images is undergoing a fundamental transformation, driven by continuous advances in 3D Gaussian Splatting (3DGS). By modeling scenes explicitly as collections of 3D Gaussians, 3DGS enables efficient rasterization through volumetric splatting, offering thus a seamless integration with common graphics pipelines. Despite its real-time rendering capabilities for novel view synthesis, 3DGS suffers from a high memory footprint, the tendency to bake lighting effects directly into its representation, and limited support for secondary-ray effects. This tutorial provides a concise yet comprehensive overview of the 3DGS pipeline, starting from its splatting formulation and then exploring the main efforts in addressing its limitations. Finally, we survey a range of applications that leverage 3DGS for surface reconstruction, avatar modeling, animation, and content generation-highlighting its efficient rendering and suitability for feed-forward pipelines.

Figures (5)

• Figure 1: Illustration of the 3DGS pipeline. Given a set of posed input images (left), a sparse point cloud from Structure-from-Motion (SfM) is used to initialize a set of colored 3D Gaussians (middle). These Gaussians are then optimized via volume splatting and can support downstream tasks such as novel view synthesis and surface reconstruction (right).
• Figure 2: Overview of the 3DGS pipeline. The process begins (left) with a set of posed images captured around an object, from which a sparse SfM point cloud is reconstructed. Gaussians are then initialized over this point cloud and optimized (center) through differentiable volumetric splatting. The rendered image is compared to the input views using a photometric loss, whose gradient is used to update the Gaussian parameters. To enhance spatial coverage and avoid under- or over-representation, 3DGS incorporates an adaptation step (right) that dynamically adds (via splitting or cloning) or removes Gaussians during training.
• Figure 3: Illustration of 2DGS huang20242d. The method represents the object as a set of 2D disks and successfully recovers both high-quality view synthesis and high-resolution normal maps. Image from Huang et al. huang20242d.
• Figure 4: Overview of avatar generation methods. These methods take full-body or head-only inputs, typically processed using body models like SMPL smpl or FLAME flame, respectively. Some approaches use feed-forward networks (FFNs) to generate Gaussians, while others employ MLPs for color effects. Additionally, some methods use text input to guide avatar generation.
• Figure 5: The yellow arrows represent heuristic-based sparse view NVS, e.g. CoR-GS zhang2024cor and MAtCha guedon2025matcha, orange represents FFNs for NVS from sparse views, single image, or video charatan2024pixelsplatchen2024mvsplatszymanowicz2024flash3dzhang2024gsye2024noposplat. Finally, purple shows diffusion models being used to generate content from all sorts of inputs tang2024lgmxu2024grmtang2023dreamgaussianchen2024larawang2024shapemotion4dreconstructionren2024l4gmwu2024cat4dcreate4dmultiview.