Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

3D-LMVIC: Learning-based Multi-View Image Coding with 3D Gaussian Geometric Priors

Yujun Huang, Bin Chen, Niu Lian, Baoyi An, Shu-Tao Xia

TL;DR

This work tackles the inefficiency of existing multi-view image compression under wide-baseline camera setups by introducing 3D-LMVIC, a learning-based framework that leverages differentiable 3D Gaussian Splatting (3D-GS) to derive depth priors for accurate disparity estimation and cross-view fusion. It couples a depth-map compression module with a 3D-aware disparity/mask mechanism and a multi-view sequence ordering strategy to maximize overlap between adjacent views, all trained with a joint rate-distortion objective. Key contributions include (i) 3D-GS based depth and disparity estimation with a robust median-depth rule, (ii) a depth-map codec that captures inter-view geometry, (iii) a distance-based view ordering to improve cross-view redundancy, and (iv) comprehensive experiments showing superior compression efficiency and improved disparity accuracy across Tanks&Temples, Mip-NeRF 360, and Deep Blending datasets. The approach yields practical impact for VR/AR and autonomous driving where large disparities and high data volumes are common, enabling more efficient storage and transmission of rich multi-view content.

Abstract

Existing multi-view image compression methods often rely on 2D projection-based similarities between views to estimate disparities. While effective for small disparities, such as those in stereo images, these methods struggle with the more complex disparities encountered in wide-baseline multi-camera systems, commonly found in virtual reality and autonomous driving applications. To address this limitation, we propose 3D-LMVIC, a novel learning-based multi-view image compression framework that leverages 3D Gaussian Splatting to derive geometric priors for accurate disparity estimation. Furthermore, we introduce a depth map compression model to minimize geometric redundancy across views, along with a multi-view sequence ordering strategy based on a defined distance measure between views to enhance correlations between adjacent views. Experimental results demonstrate that 3D-LMVIC achieves superior performance compared to both traditional and learning-based methods. Additionally, it significantly improves disparity estimation accuracy over existing two-view approaches.

3D-LMVIC: Learning-based Multi-View Image Coding with 3D Gaussian Geometric Priors

TL;DR

This work tackles the inefficiency of existing multi-view image compression under wide-baseline camera setups by introducing 3D-LMVIC, a learning-based framework that leverages differentiable 3D Gaussian Splatting (3D-GS) to derive depth priors for accurate disparity estimation and cross-view fusion. It couples a depth-map compression module with a 3D-aware disparity/mask mechanism and a multi-view sequence ordering strategy to maximize overlap between adjacent views, all trained with a joint rate-distortion objective. Key contributions include (i) 3D-GS based depth and disparity estimation with a robust median-depth rule, (ii) a depth-map codec that captures inter-view geometry, (iii) a distance-based view ordering to improve cross-view redundancy, and (iv) comprehensive experiments showing superior compression efficiency and improved disparity accuracy across Tanks&Temples, Mip-NeRF 360, and Deep Blending datasets. The approach yields practical impact for VR/AR and autonomous driving where large disparities and high data volumes are common, enabling more efficient storage and transmission of rich multi-view content.

Abstract

Existing multi-view image compression methods often rely on 2D projection-based similarities between views to estimate disparities. While effective for small disparities, such as those in stereo images, these methods struggle with the more complex disparities encountered in wide-baseline multi-camera systems, commonly found in virtual reality and autonomous driving applications. To address this limitation, we propose 3D-LMVIC, a novel learning-based multi-view image compression framework that leverages 3D Gaussian Splatting to derive geometric priors for accurate disparity estimation. Furthermore, we introduce a depth map compression model to minimize geometric redundancy across views, along with a multi-view sequence ordering strategy based on a defined distance measure between views to enhance correlations between adjacent views. Experimental results demonstrate that 3D-LMVIC achieves superior performance compared to both traditional and learning-based methods. Additionally, it significantly improves disparity estimation accuracy over existing two-view approaches.
Paper Structure (33 sections, 4 theorems, 22 equations, 10 figures, 4 tables, 1 algorithm)

This paper contains 33 sections, 4 theorems, 22 equations, 10 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introductioin
  2. Related Works
  3. Proposed Method
  4. 3D-GS Based Depth and Disparity Estimation
  5. Depth Estimation
  6. Disparity Estimation
  7. Compression Framework for Images and Depth Maps
  8. Image Compression Model
  9. Depth Map Compression Model
  10. Multi-View Sequence Ordering
  11. Training Loss
  12. Experiments
  13. Experimental Setup
  14. Experimental Results
  15. Alignment Experiments
  16. ...and 18 more sections

Key Result

Lemma 3.2

For any $u = (A, B)$ and $v = (C, D)$ as defined in Definition definition:1, the following inequality holds:

Figures (10)

  • Figure 1: Illustrations of camera systems. (a) A stereo camera configuration. (b) A wide-baseline multi-camera configuration.
  • Figure 2: Overall Pipeline.
  • Figure 3: The architecture of the proposed image compression model. 'LR' represents the Leaky ReLU activation function, 'Q' denotes the quantization operation, and 'AE'/'AD' refer to the arithmetic encoder/decoder, respectively.
  • Figure 4: Illustration of the proposed image context transfer module.
  • Figure 5: Rate-distortion curves of the proposed method compared with baselines.
  • ...and 5 more figures

Theorems & Definitions (10)

  • Definition 3.1
  • Lemma 3.2
  • proof
  • Theorem 3.3
  • proof
  • Definition 3.4
  • Lemma 3.5
  • proof
  • Theorem 3.6
  • proof