RefinedFields: Radiance Fields Refinement for Planar Scene Representations

TL;DR

RefinedFields addresses the gap in in-the-wild planar scene representations by integrating a large pre-trained prior into K-Planes via an alternating training strategy. The method alternates between fitting the K-Planes to data and refining the representation through a low-rank fine-tuning of a diffusion model (LoRA) to produce a refined $P_epsilon$ that better resembles real images, then reinitializes the next fitting with the refined representation $P_gamma$. Experiments on Real Synthetic $360^{\circ}$ and Phototourism scenes show improved novel view synthesis over the base K-Planes, with ablations confirming the value of the prior and LoRA fine-tuning on $P_gamma$. While Gaussian Splatting methods still outperform in some metrics, RefinedFields demonstrates a principled way to augment planar representations with powerful 2D priors for improved 3D rendering in unconstrained settings.

Abstract

Planar scene representations have recently witnessed increased interests for modeling scenes from images, as their lightweight planar structure enables compatibility with image-based models. Notably, K-Planes have gained particular attention as they extend planar scene representations to support in-the-wild scenes, in addition to object-level scenes. However, their visual quality has recently lagged behind that of state-of-the-art techniques. To reduce this gap, we propose RefinedFields, a method that leverages pre-trained networks to refine K-Planes scene representations via optimization guidance using an alternating training procedure. We carry out extensive experiments and verify the merit of our method on synthetic data and real tourism photo collections. RefinedFields enhances rendered scenes with richer details and improves upon its base representation on the task of novel view synthesis. Our project page can be found at https://refinedfields.github.io .

Figures (12)

• Figure 1: Qualitative Results. Given images of the Trevi fountain from Phototourism phototourism, as well as a pre-trained model stable-diffusion, our method leverages the pre-trained model and refines K-Planes with finer details that are under-represented when optimizing the same K-Planes on the images alone.
• Figure 2: Scene learning procedure. The K-Planes $\mathbf{P}_\gamma$, the MLP with trainable parameters $\alpha$, and the appearance embeddings $e_i$ are learned during scene fitting. The LoRA parameters $\phi$ as well as the decoder $D_w$ are learned during scene refining. The pre-trained U-Net is frozen. Assets in violet and underlined are intermediate results. At each iteration, new planes $\mathbf{P}_\varepsilon$ are inferred and assigned to $\mathbf{P}_\gamma$, which are then corrected by scene fitting.
• Figure 3: Case study. Qualitative results on the Lego scene from the NeRF synthetic dataset nerf showcasing the optimization progression on RefinedFields, and a comparison with the ground truth and K-Planes. The training set is constrained to 50% of its initial size for both RefinedFields and K-Planes. RefinedFields refines the K-Planes representation enabling the proper reconstruction of details in the scene. At the end of optimization, the Mean Squared Error (MSE) for RefinedFields is $3.46 \times 10^{-4}$, while the one for K-Planes is $4.36 \times 10^{-4}$.
• Figure 4: Qualitative results. Results on three scenes from Phototourism phototourism. Our method refines K-Planes and leads to richer and finer details in scene renderings.
• Figure 5: Feature planes inspection. Visualization of the $(xy)$ K-Planes feature planes during the RefinedFields optimization process (\ref{['x:insp-in-process-0']}), at the end of the RefinedFields optimization (\ref{['x:insp-refinedfields-0']}), and a comparison with vanilla K-Planes-SS (\ref{['x:insp-kplanes-0']}). Feature planes within the $(xy)$ K-Planes are picked randomly.