SANR: Scene-Aware Neural Representation for Light Field Image Compression with Rate-Distortion Optimization

TL;DR

Light field images are inherently high-dimensional, making efficient compression challenging. The authors propose SANR, which injects explicit scene priors into an implicit neural representation via a hierarchical scene modeling block and enables end-to-end rate-distortion optimization using quantization-aware training and stochastic Gumbel annealing. Key contributions include multi-scale latent scene codes and an RD objective that jointly optimizes network weights and latent codes, yielding state-of-the-art rate-distortion performance with substantial BD-rate savings. Experiments on EPFL and HCI datasets demonstrate strong improvements over HEVC and other baselines, indicating SANR’s potential for storage- and transmission-critical light field applications. Overall, SANR advances light field compression by marrying scene-aware INR modeling with principled RD optimization, delivering higher fidelity reconstructions at lower bitrates.

Abstract

Light field images capture multi-view scene information and play a crucial role in 3D scene reconstruction. However, their high-dimensional nature results in enormous data volumes, posing a significant challenge for efficient compression in practical storage and transmission scenarios. Although neural representation-based methods have shown promise in light field image compression, most approaches rely on direct coordinate-to-pixel mapping through implicit neural representation (INR), often neglecting the explicit modeling of scene structure. Moreover, they typically lack end-to-end rate-distortion optimization, limiting their compression efficiency. To address these limitations, we propose SANR, a Scene-Aware Neural Representation framework for light field image compression with end-to-end rate-distortion optimization. For scene awareness, SANR introduces a hierarchical scene modeling block that leverages multi-scale latent codes to capture intrinsic scene structures, thereby reducing the information gap between INR input coordinates and the target light field image. From a compression perspective, SANR is the first to incorporate entropy-constrained quantization-aware training (QAT) into neural representation-based light field image compression, enabling end-to-end rate-distortion optimization. Extensive experiment results demonstrate that SANR significantly outperforms state-of-the-art techniques regarding rate-distortion performance with a 65.62\% BD-rate saving against HEVC.

Figures (12)

• Figure 1: This figure shows the typical INR-based and the SANR light field image compression pipeline. For the typical method, $\mathcal{G}$ is the neural network for mapping pixel coordinates $(u,v,x,y)$ and given noise $\epsilon$ to the corresponding $(R,G,B)$ values. SANR takes the input of angle coordinate $(u, v)$ and realises the image-level mapping. The typical method omits the scene information capture and cannot reduce the information gap between the input and image signals. SANR gives the network $\mathcal{G}$ with the ability to capture scene features. During the encoding stage, the typical method first trains the network $\mathcal{G}$ to fit the target light field image and then quantizes the trained network $\mathcal{G}$. The separate rate-distortion optimization leads to suboptimal compression performance. SANR realizes the end-to-end rate-distortion optimization with the QAT and Stochastic Gumbel Annealing (SGA) yang2020improving. For decoding, both methods need to entropy decode the trained parameters of the network $\mathcal{G}$ and then execute forward propagation to reconstruct light field images.
• Figure 2: Architecture overview of the proposed SANR framework. The overall network structure consists of four hierarchical scene modeling blocks (HSMBs), forming the backbone of SANR. Each HSMB contains a scene module and a view module. The scene module encodes spatial-structural priors into a latent scene code $y_i$, which is modeled by a context model that estimates its distribution parameters (mean $\mu_i$ and standard deviation $\sigma_i$) for entropy coding. The view module transforms the intermediate feature $f_i$ into the output feature using a modulated CNN, which is conditioned on the input angular coordinate $(u, v)$ to synthesize the corresponding sub-aperture image (SAI).
• Figure 3: The architecture of the modulated CNN. The angular coordinate $(u, v)$ is used to generate modulation parameters: horizontal and vertical weights $W_u$, $W_v$ and biases $b_u$, $b_v$. These, together with spatial weights $W_s$, are applied through tensor outer products with a shared kernel base $B$. The resulting kernels are concatenated and used in convolution with the input feature. The output is obtained by adding the modulated bias $b_{u,v}$, allowing the network to dynamically generate SAI views on demand.
• Figure 4: The figure illustrates how latent scene codes progressively capture scene features during training.
• Figure 5: The average error maps for SANR without large-scale latent codes $y_3, y_4$ and without small-scale latent codes $y_1, y_2$.