A Dynamic By-example BTF Synthesis Scheme

TL;DR

This work tackles the challenge of rendering photorealistic materials by addressing the storage and pre-generation bottlenecks of measured $6D$ BTFs. It introduces a neural dimensional decomposition, the Triple Plane, which splits a $6D$ BTF into three independent $2D$ planes and a lightweight neural operator for recovery. Dynamic, by-example synthesis is performed on the $2D$ positional plane, enabling on-demand generation of infinitely large, non-repetitive BTFs without precomputation. Across UBO2014 BTFs, the approach achieves faithful high-frequency details with interactive rendering times, offering a flexible framework that can pair with various texture-synthesis methods and neural representations.

Abstract

Measured Bidirectional Texture Function (BTF) can faithfully reproduce a realistic appearance but is costly to acquire and store due to its 6D nature (2D spatial and 4D angular). Therefore, it is practical and necessary for rendering to synthesize BTFs from a small example patch. While previous methods managed to produce plausible results, we find that they seldomly take into consideration the property of being dynamic, so a BTF must be synthesized before the rendering process, resulting in limited size, costly pre-generation and storage issues. In this paper, we propose a dynamic BTF synthesis scheme, where a BTF at any position only needs to be synthesized when being queried. Our insight is that, with the recent advances in neural dimension reduction methods, a BTF can be decomposed into disjoint low-dimensional components. We can perform dynamic synthesis only on the positional dimensions, and during rendering, recover the BTF by querying and combining these low-dimensional functions with the help of a lightweight Multilayer Perceptron (MLP). Consequently, we obtain a fully dynamic 6D BTF synthesis scheme that does not require any pre-generation, which enables efficient rendering of our infinitely large and non-repetitive BTFs on the fly. We demonstrate the effectiveness of our method through various types of BTFs taken from UBO2014.

Figures (8)

• Figure 1: The direct visualization of our feature planes. The positional feature plane shows highly semantic characteristics with detailed structures and variation information close to the BTF. The resolution of each feature plane follows Biplane fan:2023:BTF's setting.
• Figure 2: Our method's pipeline. We first decompose 6D BTF into three 2D planes ($f^{(\mathbf{U})}, f^{(\mathbf{H})}$ and $f^{(\mathbf{D})}$), then perform dynamic synthesis on the positional plane as if it is a 2D texture. The synthesized reflectance is recovered via a lightweight MLP with the input of positional and directional features.
• Figure 3: Validation of our method's representation capability for different BTFs. We visualize our results along with ground truth (GT) BTFs with two different pairs of directions for each different material. We use structural dissimilarity (DSSIM$\downarrow$) as the error metric.
• Figure 4: Comparison with NeuMIP kuznetsov:2021:neumip and reference without applying synthesis, i.e., use repetitive tiling. The reference image is generated by interpolating the original BTF data. We show the FLIP error$\downarrow$ and the error image on the bottom. Triple Plane is closer to the reference with more accurate highlights and patterns, but NeuMIP is also good. Therefore, both can be used in our scheme.
• Figure 5: We validate our by-example BTF synthesis in different scales by scaling the UV coordinate, the scaling factor is marked on the bottom-right. Even on a very large scale ($45 \times$), our method faithfully maintains the accurate appearance of the BTF, which demonstrates our capability of generating an infinitely large, non-repetitive BTF.