Neural Graphics Texture Compression Supporting Random Access

TL;DR

An asymmetric auto-encoder framework that employs a convolutional encoder to capture detailed information in a bottleneck-latent space, and at decoder side a fully connected network, designed to enable random access and support many-channel texture sets.

Abstract

Advances in rendering have led to tremendous growth in texture assets, including resolution, complexity, and novel textures components, but this growth in data volume has not been matched by advances in its compression. Meanwhile Neural Image Compression (NIC) has advanced significantly and shown promising results, but the proposed methods cannot be directly adapted to neural texture compression. First, texture compression requires on-demand and real-time decoding with random access during parallel rendering (e.g. block texture decompression on GPUs). Additionally, NIC does not support multi-resolution reconstruction (mip-levels), nor does it have the ability to efficiently jointly compress different sets of texture channels. In this work, we introduce a novel approach to texture set compression that integrates traditional GPU texture representation and NIC techniques, designed to enable random access and support many-channel texture sets. To achieve this goal, we propose an asymmetric auto-encoder framework that employs a convolutional encoder to capture detailed information in a bottleneck-latent space, and at decoder side we utilize a fully connected network, whose inputs are sampled latent features plus positional information, for a given texture coordinate and mip level. This latent data is defined to enable simplified access to multi-resolution data by simply changing the scanning strides. Experimental results demonstrate that this approach provides much better results than conventional texture compression, and significant improvement over the latest method using neural networks.

Figures (10)

• Figure 1: (a) Overview of our neural texture compression (b) Bitrate comparison over ASTC in terms of PSNR. Our method significantly outperforms the state-of-the-art texture compression method, NTC Vaidyanathan2023, with a 40.8% in the BD-rate saving.
• Figure 2: Centered kernel alignment (CKA) of features stored at pyramid levels $k$ and $l$ of the feature pyramid $F_i$, i.e., $F^k_i$ and $F^l_i$, evaluated on “Ceramic roof 01” texture set, retrieved from https://polyhaven.com/.
• Figure 3: An example of texture set consisting of a diffuse map, normal map, displacement map and combined ambient occlusion (ao) and roughness map for a brick wall pattern retrieved from https://polyhaven.com.
• Figure 4: Overview of our method. Given a texture set $\mathbf{T}$, the encoder $\mathcal{E}_{\phi}$ produces a bottleneck latent representation $\mathbf{Z}_{\mathrm{SCR}}$. The constructors $\mathcal{C}_i, i=0,1$ then utilize this latent representation to construct the grid-pair $\mathbf{G}_i$. At a specific position $x,y$ and mip level $m$, the grid samplers $\mathcal{S}_i$ extract features $\mathbf{Y}_i$ from $\mathbf{G}_i$. These extracted features, along with positional encoding $\mathbf{P}$ are subsequently fed into the decoder $\mathcal{D}_{\theta}$, which reconstructs the texel $\widehat{\mathbf{T}}^m(x,y)$ at the given position and mip level.
• Figure 5: Comparison with the state-of-the-art method NTC Vaidyanathan2023.