Neural Texture Block Compression

TL;DR

A novel block texture compression method with neural networks, Neural Texture Block Compression (NTBC), which learns the mapping from uncompressed textures to block-compressed textures, which allows for significantly reduced storage costs without any change in the shaders.

Abstract

Block compression is a widely used technique to compress textures in real-time graphics applications, offering a reduction in storage size. However, their storage efficiency is constrained by the fixed compression ratio, which substantially increases storage size when hundreds of high-quality textures are required. In this paper, we propose a novel block texture compression method with neural networks, Neural Texture Block Compression (NTBC). NTBC learns the mapping from uncompressed textures to block-compressed textures, which allows for significantly reduced storage costs without any change in the shaders.Our experiments show that NTBC can achieve reasonable-quality results with up to about 70% less storage footprint, preserving real-time performance with a modest computational overhead at the texture loading phase in the graphics pipeline.

Figures (12)

• Figure 1: Naive approach. Two MLPs infer weights and endpoints. The weight network is trained to output continuous weights $\hat{w_f}$, and the endpoint network is later fine-tuned with quantized weights $w_n$. Values in yellow squares form a compressed block.
• Figure 2: Neural Texture Block Compression. Two MLPs infer endpoints and uncompressed colors. (a) In the inference procedure, we compute weight indices $n$ using predicted endpoints and colors. A resulting compressed block is indicated by yellow squares. (b) In the training procedure, two networks are trained with reference endpoints ($\mathcal{L}_e$) and uncompressed colors ($\mathcal{L}_c$), and additional errors of decoded colors ($\mathcal{L}_{cd}$). Purple indicates reference values and their losses. For optimization, we propagate the gradients through the argmax operation as if an index $n$ was computed with the softmax operation.
• Figure 3: Quantitative comparison for all materials in our dataset for different methods. Materials are retrieved from ambientCG and polyhaven. CS. and AG. are short for conservative and aggressive, respectively. Diffuse, Normal, and ARM are RGB textures compressed as BC1, while others are single-channel textures compressed as BC4.
• Figure 4: Ratio of average PSNR and SSIM of the naive approach and NTBC to the reference BC vs. their storage costs. The solid lines show the average PSNR ratio while the dashed lines show the average SSIM ratio. AG. and CS. are short for aggressive and conservative, respectively.
• Figure 5: Qualitative comparison of the naive approach and NTBC for the diffuse and displacement textures in the Carpet015 material retrieved from ambientCG ambientCG. Diffuse and displacement textures are compressed as BC1 and BC4, respectively.