COREA: Coarse-to-Fine 3D Representation Alignment Between Relightable 3D Gaussians and SDF via Bidirectional 3D-to-3D Supervision

TL;DR

COREA addresses the geometric fidelity gap in 3D Gaussian splatting by introducing a coarse-to-fine, bidirectional 3D-to-3D supervision between relightable Gaussians and an SDF. Depth- and normal-based alignment, coupled with dual-density control, enables precise 3D geometry learning while maintaining memory efficiency. The framework unifies novel-view synthesis, mesh reconstruction, and physically-based relighting under a single pipeline, outperforming prior Gaussian-based methods across NVS, PBR, and CD on standard benchmarks. This approach demonstrates the viability of direct 3D-space supervision for robust inverse rendering and high-fidelity relighting in complex scenes.

Abstract

We present COREA, the first unified framework that jointly learns relightable 3D Gaussians and a Signed Distance Field (SDF) for accurate geometry reconstruction and faithful relighting. While recent 3D Gaussian Splatting (3DGS) methods have extended toward mesh reconstruction and physically-based rendering (PBR), their geometry is still learned from 2D renderings, leading to coarse surfaces and unreliable BRDF-lighting decomposition. To address these limitations, COREA introduces a coarse-to-fine bidirectional 3D-to-3D alignment strategy that allows geometric signals to be learned directly in 3D space. Within this strategy, depth provides coarse alignment between the two representations, while depth gradients and normals refine fine-scale structure, and the resulting geometry supports stable BRDF-lighting decomposition. A density-control mechanism further stabilizes Gaussian growth, balancing geometric fidelity with memory efficiency. Experiments on standard benchmarks demonstrate that COREA achieves superior performance in novel-view synthesis, mesh reconstruction, and PBR within a unified framework.

Figures (10)

• Figure 1: Overview of COREA. (a) COREA is the first unified framework that jointly optimizes relightable 3D Gaussians and an SDF through bidirectional 3D-to-3D supervision, enabling accurate geometry learning for novel-view synthesis (NVS), mesh reconstruction, and physically-based relighting (PBR). (b) Quantitative comparison among PBR-capable methods shows that COREA achieves the highest PSNR for both NVS and PBR. (c) Among mesh-reconstructable methods, COREA attains the lowest geometric error with high rendering fidelity.
• Figure 2: Comparison of normal representations. The alpha-blended normal is a 2D normal projection that becomes blurry after mixing multiple Gaussians, resulting in coarse geometric cues. In contrast, the pixel-wise depth gradient is derived from the 3DGS depth field, preserving sharper and more structured geometry and enabling direct supervision in 3D space with SDF normals (Eq. \ref{['eq:sdf_normal']}).
• Figure 3: Overview of the COREA framework. Our method jointly trains relightable 3D Gaussians and a SDF via coarse-to-fine bidirectional 3D-to-3D supervision. The first stage, Bidirectional 3D-to-3D Supervision, consists of two complementary steps: (i) DSA aligns the SDF to the 3DGS by leveraging depth rendered from Gaussians and matching SDF normals to pixel-wise depth gradients of 3DGS (Eq. \ref{['eq:sdf_normal']}); (ii) NGA aligns 3DGS to the SDF by matching Gaussian depth to the SDF depth and supervising Gaussian normals with SDF normals (Eq. \ref{['eq:3DGS_normal']}). To prevent excessive Gaussian splitting during NGA, the DDC module suppresses unnecessary densification for efficient geometry refinement. These steps are jointly applied in each iteration to refine coarse and fine geometry. In the second stage, Inverse PBR is performed using the refined geometry to decompose BRDF and lighting and enable relighting under novel illumination conditions.
• Figure 4: Dual-Density Control (DDC). Gaussians with accumulated gradients exceeding the threshold are shown in red, while others remain blue. For those in red, splitting matrices from image and normal losses are combined into the total matrix $S_{\text{total}}$ (Eq. \ref{['eq:splitting_total']}). Only those with $\lambda_{\min}(S_{\text{total}}) < 0$ are further divided into green Gaussians, exclusively when it contributes to overall loss reduction.
• Figure 5: Qualitative Comparison Across Tasks. We compare COREA with recent Gaussian-based methods on three tasks: NVS, PBR, and Mesh Reconstruction. All results are rendered on a white background for consistent visual comparison. Artifacts such as black background patches or dark speckles observed in some baselines stem from excessive Gaussian opacity; by contrast, COREA produces clean, artifact-free renderings on white backgrounds. Our method also provides results for all three tasks, whereas others show N/S (Not Supported) or OOM (Out of Memory) in several cases. Overall, COREA yields sharper novel views, more faithful BRDF and lighting decomposition, and finer geometric details through coarse-to-fine bidirectional 3D-to-3D supervision. Blue, green, and orange boxes denote NVS, PBR, and mesh reconstruction results, respectively. Additional qualitative results are provided in the Supplementary Material and demo video.