Colorizing Monochromatic Radiance Fields

TL;DR

ColorNeRF tackles colorizing monochromatic radiance fields by switching to a representation-prediction paradigm in the Lab color space. It first learns luminance and density from monochrome images, then predicts the $ab$ color representation, using a query-based colorization module, histogram-guided purification, and a classification-based color injection to ensure plausibility and vividness. The approach achieves state-of-the-art results on synthetic and real monochrome data, enabling colorful novel-view synthesis and offering a path to rejuvenate archival monochrome footage. The work demonstrates the practical impact of integrating 2D colorization models with 3D implicit representations to recover color in a coherent, view-consistent way.

Abstract

Though Neural Radiance Fields (NeRF) can produce colorful 3D representations of the world by using a set of 2D images, such ability becomes non-existent when only monochromatic images are provided. Since color is necessary in representing the world, reproducing color from monochromatic radiance fields becomes crucial. To achieve this goal, instead of manipulating the monochromatic radiance fields directly, we consider it as a representation-prediction task in the Lab color space. By first constructing the luminance and density representation using monochromatic images, our prediction stage can recreate color representation on the basis of an image colorization module. We then reproduce a colorful implicit model through the representation of luminance, density, and color. Extensive experiments have been conducted to validate the effectiveness of our approaches. Our project page: https://liquidammonia.github.io/color-nerf.

Figures (9)

• Figure 1: With multi-view monochromatic images (a) as inputs, (b-e) are three novel views synthesised by NeRF models. Existing color editing NeRFs (b) ARF zhang2022arf and (c) CLIP-NeRF wang2022clipnerf could not guarantee pixel-wise color adherence, while using "colorize-then-fuse" solution (d) CT$^2$WengCT2+NeRF) suffers from color inconsistency across different views. The proposed ColorNeRF (e) can generate a more plausible and vivid colorized NeRF compared to previous models.
• Figure 2: The overall pipeline of the proposed ColorNeRF. With rays from multiple viewpoints as inputs, luminance and density representation is first constructed with supervision over the ground truth monochromatic images, yielding monochromatic image patches $\hat{\mathbf{L}}_\mathcal{P}$. Then we predict color with an off-the-shelf 2D colorization module $\mathcal{F}(\cdot)$, followed by our histogram-guided purification module to enhance plausibility. Lastly, we inject the color information in $\mathbf{Z}_\mathcal{P}$ to the color representation with our classification-based color injection module. The final output $\hat{\mathbf{C}}_{\mathcal{P}}$ is calculated by the concatenation of $\hat{\mathbf{L}}_{\mathcal{P}}$ and $\hat{\mathbf{Y}}_{\mathcal{P}}$, followed by the $Lab$ to RGB conversion.
• Figure 3: Our query-based injection strategy could generate plausible results from inconsistent colors. (a) is the input monochromatic image with different sampled patches (denoted as rectangles); (b) is the final output of our model with consistent color; (c) are the results from CT$^2$WengCT2, corresponding to image patches in (a).
• Figure 4: Outliers from colorized image patches. For both examples, the left is the reference image and the right is the colorized image with outliers, marked by red rectangles.
• Figure 5: One image sample for each scene in our dataset. The names of the scenes are listed below the image.