Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gaussian Splatting-based Low-Rank Tensor Representation for Multi-Dimensional Image Recovery

Yiming Zeng, Xi-Le Zhao, Wei-Hao Wu, Teng-Yu Ji, Chao Wang

TL;DR

This work addresses the limitations of tensor-SVD approaches in multi-dimensional image recovery, notably the coarse latent-tensor approximation and fixed, data-agnostic transform bases that fail to capture local high-frequency information. It introduces Gaussian Splatting-based Low-rank Tensor Representation (GSLR), which jointly models a latent tensor $\mathcal{A}$ via 2D Gaussian splatting and a transform matrix $\mathbf{T}$ via 1D Gaussian splatting, yielding a continuous representation $\mathcal{X}=\mathcal{A} \times_3 \mathbf{T}$. The framework includes a principled unsupervised recovery model with a nuclear-norm regularization on frontal slices, and demonstrates, through extensive experiments on color images and multispectral images under random, tube, and slice missing patterns, that GSLR outperforms state-of-the-art t-SVD-based methods, particularly in preserving local high-frequency details. Theoretical connections show that GSLR generalizes classical t-SVD under degenerate settings, highlighting its stronger representational capacity and practical impact for high-fidelity multi-dimensional image recovery and related tasks.

Abstract

Tensor singular value decomposition (t-SVD) is a promising tool for multi-dimensional image representation, which decomposes a multi-dimensional image into a latent tensor and an accompanying transform matrix. However, two critical limitations of t-SVD methods persist: (1) the approximation of the latent tensor (e.g., tensor factorizations) is coarse and fails to accurately capture spatial local high-frequency information; (2) The transform matrix is composed of fixed basis atoms (e.g., complex exponential atoms in DFT and cosine atoms in DCT) and cannot precisely capture local high-frequency information along the mode-3 fibers. To address these two limitations, we propose a Gaussian Splatting-based Low-rank tensor Representation (GSLR) framework, which compactly and continuously represents multi-dimensional images. Specifically, we leverage tailored 2D Gaussian splatting and 1D Gaussian splatting to generate the latent tensor and transform matrix, respectively. The 2D and 1D Gaussian splatting are indispensable and complementary under this representation framework, which enjoys a powerful representation capability, especially for local high-frequency information. To evaluate the representation ability of the proposed GSLR, we develop an unsupervised GSLR-based multi-dimensional image recovery model. Extensive experiments on multi-dimensional image recovery demonstrate that GSLR consistently outperforms state-of-the-art methods, particularly in capturing local high-frequency information.

Gaussian Splatting-based Low-Rank Tensor Representation for Multi-Dimensional Image Recovery

TL;DR

This work addresses the limitations of tensor-SVD approaches in multi-dimensional image recovery, notably the coarse latent-tensor approximation and fixed, data-agnostic transform bases that fail to capture local high-frequency information. It introduces Gaussian Splatting-based Low-rank Tensor Representation (GSLR), which jointly models a latent tensor via 2D Gaussian splatting and a transform matrix via 1D Gaussian splatting, yielding a continuous representation . The framework includes a principled unsupervised recovery model with a nuclear-norm regularization on frontal slices, and demonstrates, through extensive experiments on color images and multispectral images under random, tube, and slice missing patterns, that GSLR outperforms state-of-the-art t-SVD-based methods, particularly in preserving local high-frequency details. Theoretical connections show that GSLR generalizes classical t-SVD under degenerate settings, highlighting its stronger representational capacity and practical impact for high-fidelity multi-dimensional image recovery and related tasks.

Abstract

Tensor singular value decomposition (t-SVD) is a promising tool for multi-dimensional image representation, which decomposes a multi-dimensional image into a latent tensor and an accompanying transform matrix. However, two critical limitations of t-SVD methods persist: (1) the approximation of the latent tensor (e.g., tensor factorizations) is coarse and fails to accurately capture spatial local high-frequency information; (2) The transform matrix is composed of fixed basis atoms (e.g., complex exponential atoms in DFT and cosine atoms in DCT) and cannot precisely capture local high-frequency information along the mode-3 fibers. To address these two limitations, we propose a Gaussian Splatting-based Low-rank tensor Representation (GSLR) framework, which compactly and continuously represents multi-dimensional images. Specifically, we leverage tailored 2D Gaussian splatting and 1D Gaussian splatting to generate the latent tensor and transform matrix, respectively. The 2D and 1D Gaussian splatting are indispensable and complementary under this representation framework, which enjoys a powerful representation capability, especially for local high-frequency information. To evaluate the representation ability of the proposed GSLR, we develop an unsupervised GSLR-based multi-dimensional image recovery model. Extensive experiments on multi-dimensional image recovery demonstrate that GSLR consistently outperforms state-of-the-art methods, particularly in capturing local high-frequency information.

Paper Structure

This paper contains 19 sections, 2 theorems, 6 equations, 9 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. T-SVD Methods
  4. Gaussian Splatting
  5. Notations and Preliminaries
  6. Proposed Method
  7. Proposed GSLR
  8. 2D Gaussian Splatting-based Latent Tensor Approximation
  9. 1D Gaussian Splatting-based Transform Matrix Approximation
  10. Connection to Classical t-SVD Method
  11. Unsupervised Multi-Dimensional Image Recovery Model
  12. Experiments
  13. Experimental Settings
  14. Experimental Results
  15. Discussions
  16. ...and 4 more sections

Key Result

Lemma 1

When the parameters $\theta_{\mathcal{A}}$ and $\theta_{\mathbf{T}}$ of the 2D Gaussian splatting and the 1D Gaussian splatting satisfy: then the 2D and 1D Gaussian splatting can explicitly represent arbitrary latent tensor $\mathcal{A}$ and arbitrary transform matrix $\mathbf{T}$.

Figures (9)

  • Figure 1: Comparison of different t-SVD methods for multi-dimensional image recovery under three missing patterns. The first four columns display the pseudo-color images reconstructed by TNN TNN, HLRTF HLRTF, the proposed GSLR, and the ground truth (GT), respectively, while the last column shows the recovered spectral curves at the spatial location $(180, 130)$. The results clearly demonstrate that GSLR produces reconstructions most consistent with the GT, effectively recovering local high-frequency details in both the spatial dimensions and the mode-3 fibers.
  • Figure 2: Overall framework of the GSLR-based multi-dimensional image recovery model. The input is the incomplete observed image $\mathcal{O} \in \mathbb{R}^{H\times W\times B}$. GSLR employs tailored 2D Gaussian splatting and 1D Gaussian splatting to generate the latent tensor $\mathcal{A}\in \mathbb{R}^{H\times W\times R}$ and accompanying transform matrix $\mathbf{T}\in \mathbb{R}^{B\times R}$, respectively. By applying the transform $\mathbf{T}$ to the latent tensor $\mathcal{A}$, we obtain the recovered image $\mathcal{X} \in \mathbb{R}^{H\times W\times B}$.
  • Figure 3: The framework of the 2D Gaussian splatting-based latent tensor approximation.
  • Figure 4: The framework of the 1D Gaussian splatting-based transform matrix approximation.
  • Figure 5: Reconstructed results and zoomed-in details by different methods under the random missing (SR = 0.10).
  • ...and 4 more figures

Theorems & Definitions (2)

  • Lemma 1
  • Theorem 1