Real-time Neural Woven Fabric Rendering

TL;DR

A lightweight neural network to represent different types of woven fabrics at different scales is proposed, which achieves rendering and editing woven fabrics at nearly 60 frames per second on an RTX 3090, showing a quality close to the ground truth and being free from visible aliasing and noise.

Abstract

Woven fabrics are widely used in applications of realistic rendering, where real-time capability is also essential. However, rendering realistic woven fabrics in real time is challenging due to their complex structure and optical appearance, which cause aliasing and noise without many samples. The core of this issue is a multi-scale representation of the fabric shading model, which allows for a fast range query. Some previous neural methods deal with the issue at the cost of training on each material, which limits their practicality. In this paper, we propose a lightweight neural network to represent different types of woven fabrics at different scales. Thanks to the regularity and repetitiveness of woven fabric patterns, our network can encode fabric patterns and parameters as a small latent vector, which is later interpreted by a small decoder, enabling the representation of different types of fabrics. By applying the pixel's footprint as input, our network achieves multi-scale representation. Moreover, our network is fast and occupies little storage because of its lightweight structure. As a result, our method achieves rendering and editing woven fabrics at nearly 60 frames per second on an RTX 3090, showing a quality close to the ground truth and being free from visible aliasing and noise.

Figures (8)

• Figure 1: Separation of BSDF distributions. We separate the BSDF value into several components, considering specular/diffuse and yarn type (weft/warp). Thanks to this separation, the distributions become much simpler. The BSDF or components are visualized by mapping the incoming directions to the horizontal axis and the outgoing direction to the vertical axis. Here, we use a plain pattern with white color and compute its BSDF/component values by Monte-Carlo point sampling within a patch with the query size set as $205 \times 205$.
• Figure 2: The structure of our neural network. Our network consists of an encoder and a decoder. The encoder takes the input of fabric patterns (normal and orientation textures) and parameters (roughness and height field scaling). These inputs are encoded into a material latent vector by the encoder, which consists of a modified ResNet He:2016:Resnet and an MLP. Then, the material latent vector is fused with the spatial query ($\mathcal{P}$) first, concatenated with the angular inputs ($\omega_{\mathrm{i}}, \omega_{\mathrm{o}}$) and then fed to the angular decoder to get four components. Both the encoder and decoder include a residual block composed of two fully connected layers and a skip connection before the last leaky ReLU function.
• Figure 3: Comparison among our method, NeuMIP Kuznetsov:2021:NeuMip and modified Jin et al. Jin:2022:inverse on seen materials. While NeuMIP produces the lowest MSE occasionally, it has to be trained per material and does not support editing. The results by modified Jin et al. Jin:2022:inverse show noticeable aliasing, even with more samples. In contrast, our results are much smoother and free from aliasing.
• Figure 4: Comparison between our method and modified Jin et al. Jin:2022:inverse on an unseen material (a twill $4 \times 4$ pattern). Our method produces results with lower MSE and less aliasing than both rendering results (1 SPP and 3 SPP) by modified Jin et al. Jin:2022:inverse.
• Figure 5: Comparison between our method and modified Jin et al. Jin:2022:inverse on an unseen BTDF material (a plain pattern). Both rendering results (with 1 SPP and 3 SPP) by modified Jin et al. Jin:2022:inverse have apparent aliasing, while our results are closer to the ground truth.