Cinematic Gaussians: Real-Time HDR Radiance Fields with Depth of Field

TL;DR

A lightweight method based on 3D Gaussian Splatting that utilizes multi‐view LDR images of a scene with varying exposure times, apertures, and focus distances as input to reconstruct a high‐dynamic‐range (HDR) radiance field and demonstrates that the combined treatment of HDR and depth of field facilitates real‐time cinematic rendering, outperforming the state of the art.

Abstract

Radiance field methods represent the state of the art in reconstructing complex scenes from multi-view photos. However, these reconstructions often suffer from one or both of the following limitations: First, they typically represent scenes in low dynamic range (LDR), which restricts their use to evenly lit environments and hinders immersive viewing experiences. Secondly, their reliance on a pinhole camera model, assuming all scene elements are in focus in the input images, presents practical challenges and complicates refocusing during novel-view synthesis. Addressing these limitations, we present a lightweight method based on 3D Gaussian Splatting that utilizes multi-view LDR images of a scene with varying exposure times, apertures, and focus distances as input to reconstruct a high-dynamic-range (HDR) radiance field. By incorporating analytical convolutions of Gaussians based on a thin-lens camera model as well as a tonemapping module, our reconstructions enable the rendering of HDR content with flexible refocusing capabilities. We demonstrate that our combined treatment of HDR and depth of field facilitates real-time cinematic rendering, outperforming the state of the art.

Figures (8)

• Figure 3: Comparison of all-in-focus reconstruction. Our method provides better or comparable results to Deblur-Splatting method.
• Figure 4: Comparison of HDR reconstruction. Left-most column presents input LDR images for a given view. The following columns depict results acquired by HDR-NeRF, Ground Truth, and results of our method, respectively. Please note, that our method properly reconstructs the shape of the flame and reproduces successfully high-frequency content, while these details are missing in the results generated by HDR-NeRF. To depict the accuracy of both methods we provide the HDR-VDP-3 error map in the top right corner of the images. To better show the differences in the highlights, we render these regions at -3 stops, as shown in the yellow box.
• Figure 5: All-in-focus HDR reconstruction. Our method can successfully reconstruct sharp HDR images from defocused images captured with different exposure times.
• Figure 6: Post-editing results. The leftmost column shows the selected input views. The 3 $\times$ 3 grid to the right displays novel views, with each row illustrating the manipulation capabilities of one parameter: focus distance, aperture size, and exposure.
• Figure 7: The defocus loss provides higher quality reconstruction for high-frequency patterns.