DoF-Gaussian: Controllable Depth-of-Field for 3D Gaussian Splatting

TL;DR

DoF-Gaussian tackles the limitation of 3D Gaussian Splatting when input imagery exhibits shallow depth-of-field by introducing a lens-based imaging model with learnable aperture $A$ and focus $F$. It adds per-scene depth priors and a defocus-to-focus adaptation to match real circle-of-confusion and improve depth accuracy, supported by a synthetic dataset to evaluate refocusing and lens parameter learning. Empirical results show improved defocus deblurring and higher-quality novel-view synthesis compared with baselines, along with interactive DoF control and CoC customization. The work advances controllable DoF in 3D scene representations and suggests a path toward broader applicability under real-world imaging conditions.

Abstract

Recent advances in 3D Gaussian Splatting (3D-GS) have shown remarkable success in representing 3D scenes and generating high-quality, novel views in real-time. However, 3D-GS and its variants assume that input images are captured based on pinhole imaging and are fully in focus. This assumption limits their applicability, as real-world images often feature shallow depth-of-field (DoF). In this paper, we introduce DoF-Gaussian, a controllable depth-of-field method for 3D-GS. We develop a lens-based imaging model based on geometric optics principles to control DoF effects. To ensure accurate scene geometry, we incorporate depth priors adjusted per scene, and we apply defocus-to-focus adaptation to minimize the gap in the circle of confusion. We also introduce a synthetic dataset to assess refocusing capabilities and the model's ability to learn precise lens parameters. Our framework is customizable and supports various interactive applications. Extensive experiments confirm the effectiveness of our method. Our project is available at https://dof-gaussian.github.io.

Figures (11)

• Figure 1: Given a set of multi-view input images with shallow DoF, DoF-Gaussian can reconstruct a 3D-GS representation of a sharp scene. Thanks to our lens-based design, we can also achieve controllable DoF effects for a variety of applications. The example input images are taken from wu2022dof for illustration purposes. (Zoom-in for best view)
• Figure 2: Overview of DoF-Gaussian. Given input images $I$ with shallow DoF , we first apply SfM from COLMAP to obtain sparse depth $D_{sparse}$, which is used to train a depth network to derive per-scene depth priors $D_{pred}$. We then employ $D_{pred}$ to regularize the Gaussians rendered depth map $D$. Next, by developing a lens imaging model, we can render defocused images $C^*$ to simulate input images. To minimize the discrepancy in CoC, we propose an adaptation using the weight map. Finally, we can render fully clear images for novel view synthesis and achieve various effects by our controllable DoF framework.
• Figure 3: The difference between the pinhole-based model ($\mathrm{I}$) and the lens-based model ($\mathrm{II}$). For ($\mathrm{I}$), light emitted from spatial point $O$ directly hits the point $O'$ of the image plane. For ($\mathrm{II}$), we show the case that light emitted from $Q$ converges on the point $Q'$ of the image plane (In-Focus), and the case that light emitted from $P$ converges on the point $P'$ and continues to scatter onto the image plane forming a circle of confusion (Out-of-Focus).
• Figure 4: Effects of the depth priors. The per-scene adjustment of depth priors enhance the geometric structure and yield a more accurate depth map than that estimated by BAGS peng2024bags and a variant of our method without depth priors.
• Figure 5: Qualitative comparisons against all baselines. Compared to other state-of-the-art methods, our method represents sharper scenes and generates novel view images with less blur.