IReNe: Instant Recoloring of Neural Radiance Fields

TL;DR

IReNe addresses the challenge of editing color in Neural Radiance Fields with near real-time feedback by retraining only the last layer of the color MLP and introducing a trainable 3D segmentation to constrain edits to targeted regions. A key contribution is automatically classifying last-layer neurons into view-dependent and diffuse types, freezing the former to preserve view-dependent shading while fine-tuning the latter to propagate color edits consistently across views. A lightweight 3D segmentation module enables boundary-aware edits while maintaining speed, with convergence typically under 5 seconds. The approach yields significant speedups (5x–500x) and improved boundary fidelity over state-of-the-art recoloring methods, demonstrated on a new dataset with edited colors across multiple NeRF scenes, enabling interactive editing pipelines.

Abstract

Advances in NERFs have allowed for 3D scene reconstructions and novel view synthesis. Yet, efficiently editing these representations while retaining photorealism is an emerging challenge. Recent methods face three primary limitations: they're slow for interactive use, lack precision at object boundaries, and struggle to ensure multi-view consistency. We introduce IReNe to address these limitations, enabling swift, near real-time color editing in NeRF. Leveraging a pre-trained NeRF model and a single training image with user-applied color edits, IReNe swiftly adjusts network parameters in seconds. This adjustment allows the model to generate new scene views, accurately representing the color changes from the training image while also controlling object boundaries and view-specific effects. Object boundary control is achieved by integrating a trainable segmentation module into the model. The process gains efficiency by retraining only the weights of the last network layer. We observed that neurons in this layer can be classified into those responsible for view-dependent appearance and those contributing to diffuse appearance. We introduce an automated classification approach to identify these neuron types and exclusively fine-tune the weights of the diffuse neurons. This further accelerates training and ensures consistent color edits across different views. A thorough validation on a new dataset, with edited object colors, shows significant quantitative and qualitative advancements over competitors, accelerating speeds by 5x to 500x.

Figures (7)

• Figure 1: IReNe enables instant 360$^\circ$ recoloring of pre-trained NeRFs using only a single image edited by the user (top row). We introduce an optimization scheme to avoid color bleeding at object boundaries and ensure consistency in view-dependent effects. Furthermore, as illustrated in the bottom row, various types of recoloring are possible, including full-object, partial-object, and multiple-object recoloring.
• Figure 2: Overview of IReNe. We use a pre-trained NeRF and a user-edited training image $\textrm{I}^{\textrm{edit}}$. Our pre-trained NeRF is an Instant NGP mueller2022instant with: a density $\textrm{MLP}_\sigma$, with multiresolution hash features $f$, and a color $\textrm{MLP}_c$. Mapping the user's edits into the NeRF involves the following steps: 1) Automatic detection of the diffuse neurons in the last layer of $\textrm{MLP}_c$. 2) Training an $\textrm{MLP}_s$, ruled by the features $f$, to estimate a volumetric soft-segmentation $\alpha$ of the edited region. 3) Fine-tuning the weights of the diffuse neurons in the last layer of $\textrm{MLP}_c$. 4) Alpha blending with the mask $\alpha$, to estimate the color of a 3D point $\textbf{x}$ as a linear combination of the color $c_{\textbf{x}}$ predicted by the frozen weights with the color $c^{\prime}_{\textbf{x}}$ predicted by the retrained weights. 5) Volumetric rendering to obtain the edited image $\textrm{I}^\textrm{render}$. $\textrm{MLP}_s$ and the trainable last-layer neurons are optimized through standard RGB loss computation between $\textrm{I}^\textrm{render}$ and $\textrm{I}^{\textrm{edit}}$ in under 5 seconds.
• Figure 3: Volumetric rendering for the activations of 3 neurons in the last layer. Points with similar color in 3d space will share a similar activation pattern.
• Figure 4: Rendered activations of the same pose while varying the view directional encoding. Top, diffuse neuron. Bottom, view-dependent neuron.
• Figure 5: Qualitative comparison with state of the art methods. For each approach we show as a small overlayed image the input that the method requires from the user. For PaletteNeRF we show the original and the edited color palettes of the image. For IReNe, we show the region the user selects using Photoshop (or any similar editing tool). On that region we can interactively perform several color edits by modifying the HSV color within the region.