Exploring Seasonal Variability in the Context of Neural Radiance Fields for 3D Reconstruction on Satellite Imagery

TL;DR

Planet-NeRF, an extension to Sat-NeRF capable of incorporating seasonal variability through a set of month embedding vectors is proposed, revealing that Planet-NeRF outperforms prior models in the case where seasonal changes are present.

Abstract

In this work, the seasonal predictive capabilities of Neural Radiance Fields (NeRF) applied to satellite images are investigated. Focusing on the utilization of satellite data, the study explores how Sat-NeRF, a novel approach in computer vision, performs in predicting seasonal variations across different months. Through comprehensive analysis and visualization, the study examines the model's ability to capture and predict seasonal changes, highlighting specific challenges and strengths. Results showcase the impact of the sun direction on predictions, revealing nuanced details in seasonal transitions, such as snow cover, color accuracy, and texture representation in different landscapes. Given these results, we propose Planet-NeRF, an extension to Sat-NeRF capable of incorporating seasonal variability through a set of month embedding vectors. Comparative evaluations reveal that Planet-NeRF outperforms prior models in the case where seasonal changes are present. The extensive evaluation combined with the proposed method offers promising avenues for future research in this domain.

Figures (8)

• Figure 1: The Planet-NeRF network is formed by integrating the Sat-NeRF network architecture with an additional layer ($\text{MLP}_{\text{season}}$) representing the latent month embedding vectors $\textbf{e}_m\in\mathbb{R}^K$, where $m \in [1,12]$, used to predict a seasonal albedo color $\textbf{c}_m\in\mathbb{R}^{N_s}$. In contrast to Sat-NeRF, Planet-NeRF also employs positional encoding of the input. $\textbf{t}_j\in\mathbb{R}^4$ is an embedding vector representing transient objects for image $j$, $\sigma$ is the volume density, $\textbf{d}_\text{sun}\in\mathbb{R}^{N_s}$ the sun direction, $s$ the shading scalar, $\beta$ the uncertainty coefficient, $\textbf{c}_\text{a}\in\mathbb{R}^{N_s}$ the albedo prediction, and $\textbf{a}_\text{sky}\in\mathbb{R}^{N_s}$ the ambient sky color prediction. $\textbf{h}\in\mathbb{R}^{512}$ and $x\in\mathbb{R}^{N_s}$. $N_s$ denotes the number of samples along each ray. $N_s=64$ and the dimensions of $\textbf{h}$ and $\textbf{t}_j$ are the default values of Sat-NeRF.
• Figure 2: Albedo predictions $\textbf{c}_\text{a}$ for OMA_132 for models SN: Sat-NeRF, ME: Sat-NeRF + month embedding, PE: Sat-NeRF + positional encoding, and PN: Planet-NeRF.
• Figure 3: Ground truth $\textbf{c}_\text{gt}$, final RGB prediction $\textbf{c}$, and albedo $\textbf{c}_\text{a}$ predictions by Sat-NeRF for the Jacksonville areas.
• Figure 4: Ground truth $\textbf{c}_{\text{gt}}$, shading scalar $s$, and ambient sky color predictions $\textbf{a}_\text{sky}$ for Sat-NeRF on the different Jacksonville areas (for two different runs).
• Figure 5: Ground truth $\textbf{c}_{\text{gt}}$, shading scalar $s$, and ambient sky color predictions $\textbf{a}_\text{sky}$ for Sat-NeRF and Planet-NeRF on three different Omaha areas during two distinct seasons.