Real-Time Neural Materials using Block-Compressed Features

TL;DR

The paper tackles real-time material rendering by replacing dense texture data with block-compressed neural features that are decoded in a shader. It introduces block-based neural features compatible with BC6 compression, coupled with a small MLP decoder, enabling continuous, scale-aware material reconstruction directly from UV coordinates without post-filtering. Key contributions include a differentiable BC6 decompression training regime, hardware-compatible filtering via mipmapped neural textures, and a practical pipeline demonstrating memory efficiency and real-time performance on consumer GPUs. The approach achieves sharper results than traditional BC/ASTC methods at similar resolutions and opens avenues for higher-fidelity neural materials in interactive applications while maintaining low overhead.

Abstract

Neural materials typically consist of a collection of neural features along with a decoder network. The main challenge in integrating such models in real-time rendering pipelines lies in the large size required to store their features in GPU memory and the complexity of evaluating the network efficiently. We present a neural material model whose features and decoder are specifically designed to be used in real-time rendering pipelines. Our framework leverages hardware-based block compression (BC) texture formats to store the learned features and trains the model to output the material information continuously in space and scale. To achieve this, we organize the features in a block-based manner and emulate BC6 decompression during training, making it possible to export them as regular BC6 textures. This structure allows us to use high resolution features while maintaining a low memory footprint. Consequently, this enhances our model's overall capability, enabling the use of a lightweight and simple decoder architecture that can be evaluated directly in a shader. Furthermore, since the learned features can be decoded continuously, it allows for random uv sampling and smooth transition between scales without needing any subsequent filtering. As a result, our neural material has a small memory footprint, can be decoded extremely fast adding a minimal computational overhead to the rendering pipeline.

Figures (14)

• Figure 1: Block Compression algorithms encode a block of $4 \times 4$ pixels with a set of endpoints forming one or multiple line segments and index each pixel based on it's projected position in the RGB space. In the case where two or more line segments are stored, the pixels are separated into groups according to a pre-defined partition $P$.
• Figure 2: Overview of our neural material framework. (a) The neural features $T_{\theta_i}$ and the MLP $f_\eta$ are fitted through backpropagation to match the filtered material $\mathcal{F}(M)$. (b) After training, the neural features $T_{\theta_i}$ are exported as mipmapped texture sets that can be sampled by the engine and the weights $\eta$ of the MLP are exported as a binary buffer. A shader is used to perform the MLP inference after trilinearly sampling the neural texture, outputting the filtered material $\mathcal{G}_{\theta, \eta}(u, v, s) \approx \mathcal{F}(M)(u,v,s)$. Finally, The renderer can perform the shading step as usual.
• Figure 3: Neural material reconstructed from (a) raw unconstrained neural features and (b) compressed ones. The naive compression of the neural features will lead to artifacts severely affecting the visual quality.
• Figure 4: Evaluating $T_{\theta_i}$ at position $(u, v)$ and scale $s$ is done by performing a BC6 decompression then filtering the result with respect to the given scale. Our simulated BC6 decoding makes it possible to backpropagate through this operation and train the per-block neural features.
• Figure 5: When sampling a feature at a scale level $s$, the block compressed features are filtered using a trilinear interpolation. This requires sampling the two closest mip levels $\lfloor s \rfloor$ and $\lfloor s \rfloor + 1$ using a bilinear interpolation than linearly mixing the results.