LAENeRF: Local Appearance Editing for Neural Radiance Fields

TL;DR

LAENeRF is a unified framework for photorealistic and non-photorealistic appearance editing of NeRFs that enables recoloring and stylization while keeping processing time low and surpasses baseline methods both quanti-tatively and qualitatively.

Abstract

Due to the omnipresence of Neural Radiance Fields (NeRFs), the interest towards editable implicit 3D representations has surged over the last years. However, editing implicit or hybrid representations as used for NeRFs is difficult due to the entanglement of appearance and geometry encoded in the model parameters. Despite these challenges, recent research has shown first promising steps towards photorealistic and non-photorealistic appearance edits. The main open issues of related work include limited interactivity, a lack of support for local edits and large memory requirements, rendering them less useful in practice. We address these limitations with LAENeRF, a unified framework for photorealistic and non-photorealistic appearance editing of NeRFs. To tackle local editing, we leverage a voxel grid as starting point for region selection. We learn a mapping from expected ray terminations to final output color, which can optionally be supervised by a style loss, resulting in a framework which can perform photorealistic and non-photorealistic appearance editing of selected regions. Relying on a single point per ray for our mapping, we limit memory requirements and enable fast optimization. To guarantee interactivity, we compose the output color using a set of learned, modifiable base colors, composed with additive layer mixing. Compared to concurrent work, LAENeRF enables recoloring and stylization while keeping processing time low. Furthermore, we demonstrate that our approach surpasses baseline methods both quantitatively and qualitatively.

Figures (18)

• Figure 1: We propose LAENeRF, a method for Local Appearance Editing of Neural Radiance Fields. LAENeRF enables appearance edits of arbitrary content in 3D scenes while minimizing background artefacts. For a specified selection, we learn a mapping from estimated ray termination to output colors via a palette-based formulation, which may be supervised by a style loss. In this way, we elegantly combine photorealistic recoloring and non-photorealistic stylization of arbitrary content represented by a radiance field in an interactive framework.
• Figure 2: Overview of LAENeRF: Given the estimated ray terminations ${\boldsymbol{x}}_{\text{term}}$ for a region specified by $\bm{\mathcal{E}}$, we learn a mapping from positions to weights ${\hat{{\boldsymbol{w}}}}$ and offsets ${\hat{\bm{\delta}}}$. We compose the color ${\hat{{\boldsymbol{c}}}}$ using these latent outputs and a learnable color palette ${\hat{\bm{\mathcal{P}}}}$ and supervise with a content loss ${\mathcal{L}_{\text{content}}}$ and optional style losses ${\mathcal{L}_{\text{style}}}, {\mathcal{L}_{\text{TV}}}, {\mathcal{L}_{\text{depth-disc}}}$ to obtain a unified approach which supports recoloring and stylization.
• Figure 3: Illustration of our proposed distance-based palette interpolation scheme: We calculate distance weights ${d_{\text{trans}}}$ based on the per-point distance from the edit grid $\bm{\mathcal{E}}$ to the growing grid ${\bm{\mathcal{G}}}$. When constructing the modified training dataset, we interpolate between learned palette ${\hat{\bm{\mathcal{P}}}}$ and modified palette ${\hat{\bm{\mathcal{P}}}}_{\text{mod}}$ using ${d_{\text{trans}}}$.
• Figure 4: Qualitative comparison of our method with related work on the Horns scene of the LLFF dataset mildenhall2019llff. LAENeRF introduces far fewer artefacts compared to previous methods.
• Figure 5: Qualitative comparison of our method to PaletteNeRF on the mip-NeRF 360 dataset Barron2022MipNeRF360 for small-scale edits. The top-row detailed view shows the selected region for recoloring, whereas the bottom-row view shows a background region. Our method introduces fewer errors in the background whilst recoloring the selected object faithfully.