VENI: Variational Encoder for Natural Illumination

TL;DR

VENI addresses inverse rendering under natural illumination by introducing a rotation-equivariant variational autoencoder for spherical illumination. It combines a novel SO(2)-equivariant extension to Vector Neurons with a VN-ViT encoder and a rotation-equivariant neural field decoder to learn a smooth, unique latent space, enabling realistic HDR reconstructions and semantically meaningful latent interpolations. The method achieves improved reconstruction quality and latent-space structure over prior approaches like RENI++, while scaling to larger datasets due to the encoder–decoder design and variational regularization. This work advances illumination priors for inverse rendering, relighting, and related vision tasks by providing a principled, geometry-aware framework that respects the spherical nature of illumination environments.

Abstract

Inverse rendering is an ill-posed problem, but priors like illumination priors, can simplify it. Existing work either disregards the spherical and rotation-equivariant nature of illumination environments or does not provide a well-behaved latent space. We propose a rotation-equivariant variational autoencoder that models natural illumination on the sphere without relying on 2D projections. To preserve the SO(2)-equivariance of environment maps, we use a novel Vector Neuron Vision Transformer (VN-ViT) as encoder and a rotation-equivariant conditional neural field as decoder. In the encoder, we reduce the equivariance from SO(3) to SO(2) using a novel SO(2)-equivariant fully connected layer, an extension of Vector Neurons. We show that our SO(2)-equivariant fully connected layer outperforms standard Vector Neurons when used in our SO(2)-equivariant model. Compared to previous methods, our variational autoencoder enables smoother interpolation in latent space and offers a more well-behaved latent space.

Figures (13)

• Figure 1: We build a rotation-equivariant variational autoencoder model to address the limitations of the current state of-the-art illumination prior (RENI++). Our model produces a better-behaved latent space, which we demonstrate by evaluating its uniqueness through optimization of random latent codes and its reconstruction consistency via the correlation between latent space and image space distances.
• Figure 2: Our model is rotation-equivariant, meaning that a rotation of the input environment map leads to a corresponding rotation in latent space and in the reconstructed environment map.
• Figure 3: Model Overview: We adapt the Vision Transformer architecture to SO(2) equivariance on spherical images. Splitting the 360° spherical image into vertical stripes and embedding the direction vectors and color values of the pixels in each patch using an SO(2)-equivariant projection. Then we feed the sequence into a Vector Neuron-Transformer. We use a class token that is learnable in non-equivariant dimensions and zero in equivariant dimensions. The output is projected to $\mu$ and $\log(\sigma^2)$ and reparameterized. The output latent code is fed into the RENI++ decoder.
• Figure 4: Vertical stripes as patches is a valid SO(2)-equivariant patching strategy. Since we use a rotation equivariant transformer encoder, a rotation by a multiple of the patch width results in the output of the encoder being rotated in the same way.
• Figure 5: Interpolations using our model (with direct latent optimization) and RENI++ with different latent sizes. Three image pairs are interpolated: snow-forest, lake-field, mountain-road.