NeuBTF: Neural fields for BTF encoding and transfer

TL;DR

This paper proposes a novel neural material representation which jointly tackles the problems of BTF compression, tiling, and extrapolation, and demonstrates its generality and capacity to learn to represent many optical properties.

Abstract

Neural material representations are becoming a popular way to represent materials for rendering. They are more expressive than analytic models and occupy less memory than tabulated BTFs. However, existing neural materials are immutable, meaning that their output for a certain query of UVs, camera, and light vector is fixed once they are trained. While this is practical when there is no need to edit the material, it can become very limiting when the fragment of the material used for training is too small or not tileable, which frequently happens when the material has been captured with a gonioreflectometer. In this paper, we propose a novel neural material representation which jointly tackles the problems of BTF compression, tiling, and extrapolation. At test time, our method uses a guidance image as input to condition the neural BTF to the structural features of this input image. Then, the neural BTF can be queried as a regular BTF using UVs, camera, and light vectors. Every component in our framework is purposefully designed to maximize BTF encoding quality at minimal parameter count and computational complexity, achieving competitive compression rates compared with previous work. We demonstrate the results of our method on a variety of synthetic and captured materials, showing its generality and capacity to learn to represent many optical properties.

Figures (7)

• Figure 1: An overview of our neural BTF inference and training processes. Top-Inference: Using a guidance image $\mathcal{G} \in \mathbb{R}^{H \times W \times 3}$, we use our trained autoencoder to generate the Neural Texture $\mathcal{A}(\mathcal{G}) = \mathcal{T}_\mathcal{G} \in \mathbb{R}^{H \times W \times D}$, which preserves the spatial resolution of the input image but represents a higher-dimensional learned representation. This Neural Texture $\mathcal{T}_\mathcal{G}$, along with the trained renderer $\mathcal{R}$, can be queried as a regular BTF, using UVs, and target camera and light positions for regular rendering. Bottom-Train: During training, following previous work on photometric data augmentation rodriguezpardo2021transfer, we randomly select an input view$V_{\tilde{\omega}_o, \tilde{\omega}_i}$ and a target $V_{\omega_o, \omega_i}$. This allows the model to generalize to novel light or camera conditions and acts as a regularizer. To both views, we apply random rescale and cropping. Then, only to $\text{V}_{\tilde{\omega}_o, \tilde{\omega}_i}$, we randomly apply hue variations, gaussian blur, and noise, and feed it to the autoencoder, which returns a 2D latent representation of the material. A fully-convolutional decoder with sinusoidal activations receives both this latent space and the target $\omega_o, \omega_i$ camera and light angles, and estimates $\hat{V}_{\omega_o, \omega_i}$. This output is compared with $V_{\omega_o, \omega_i}$ using a multifaceted loss function.
• Figure 2: Some tileable neural materials achieved with our method. On the top row, we show a slice of the BTF used to train a NeuBTF representation. With the tileable guidance images shown on the second row, we propagate the neural texture using our autoencoders. These neural textures can be rendered to generate realistic images (third). We provide closeups on the bottom row. In the cases where the guidance image covers a larger area than the training crop, we highlight the training surface area as a green inset.
• Figure 3: Qualitative results on a variety of BTFs, from different sources. From left to right, we show results on a material from the UBO weinmann2014material BTF dataset, the UTIA haindl12ERCIM BTF dataset and two synthetic materials rendered from Substance SVBRDFs.
• Figure 4: A selection of latent channels learned by NeuBTF for a variety of materials. We use a colorspace to help visualization. Without explicit supervision, the model internally learns semantically meaningful latent spaces. For instance, in the second example on the left, the two leftmost latent spaces encode geometry, while the other two encode the two distinct colors of the printed pattern over the yarns.
• Figure 5: A failure case of our method. Compared to NeuMIP kuznetsov2021neumip, which explictly models parallax effects, our model struggles to accurately encode materials with strong displacements, as this synthetic cable knit from Substance3D. For this type of materials, NeuBTF accurately encodes orthogonal viewing angles (top row), however, it struggles at grazing angles (bottom row).