Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisiting Nonlocal Self-Similarity from Continuous Representation

Yisi Luo, Xile Zhao, Deyu Meng

TL;DR

This work addresses the limitation of traditional NSS methods to meshgrid data by introducing CRNL, a continuous representation-based nonlocal framework. CRNL uses an implicit neural representation to learn a continuous data function, segments the space into basic cubes, and forms continuous groups whose nonlocal structure is captured by a coupled low-rank function factorization with shared factor functions and group-specific cores. The approach achieves state-of-the-art performance on both on-meshgrid tasks (image inpainting and denoising) and off-meshgrid tasks (climate data prediction and point-cloud recovery) with favorable computational efficiency. By unifying similarity measurement across data types and leveraging cross-group correlations, CRNL offers a versatile and scalable tool for multi-dimensional data recovery and analysis with broad practical impact.

Abstract

Nonlocal self-similarity (NSS) is an important prior that has been successfully applied in multi-dimensional data processing tasks, e.g., image and video recovery. However, existing NSS-based methods are solely suitable for meshgrid data such as images and videos, but are not suitable for emerging off-meshgrid data, e.g., point cloud and climate data. In this work, we revisit the NSS from the continuous representation perspective and propose a novel Continuous Representation-based NonLocal method (termed as CRNL), which has two innovative features as compared with classical nonlocal methods. First, based on the continuous representation, our CRNL unifies the measure of self-similarity for on-meshgrid and off-meshgrid data and thus is naturally suitable for both of them. Second, the nonlocal continuous groups can be more compactly and efficiently represented by the coupled low-rank function factorization, which simultaneously exploits the similarity within each group and across different groups, while classical nonlocal methods neglect the similarity across groups. This elaborately designed coupled mechanism allows our method to enjoy favorable performance over conventional NSS methods in terms of both effectiveness and efficiency. Extensive multi-dimensional data processing experiments on-meshgrid (e.g., image inpainting and image denoising) and off-meshgrid (e.g., climate data prediction and point cloud recovery) validate the versatility, effectiveness, and efficiency of our CRNL as compared with state-of-the-art methods.

Revisiting Nonlocal Self-Similarity from Continuous Representation

TL;DR

This work addresses the limitation of traditional NSS methods to meshgrid data by introducing CRNL, a continuous representation-based nonlocal framework. CRNL uses an implicit neural representation to learn a continuous data function, segments the space into basic cubes, and forms continuous groups whose nonlocal structure is captured by a coupled low-rank function factorization with shared factor functions and group-specific cores. The approach achieves state-of-the-art performance on both on-meshgrid tasks (image inpainting and denoising) and off-meshgrid tasks (climate data prediction and point-cloud recovery) with favorable computational efficiency. By unifying similarity measurement across data types and leveraging cross-group correlations, CRNL offers a versatile and scalable tool for multi-dimensional data recovery and analysis with broad practical impact.

Abstract

Nonlocal self-similarity (NSS) is an important prior that has been successfully applied in multi-dimensional data processing tasks, e.g., image and video recovery. However, existing NSS-based methods are solely suitable for meshgrid data such as images and videos, but are not suitable for emerging off-meshgrid data, e.g., point cloud and climate data. In this work, we revisit the NSS from the continuous representation perspective and propose a novel Continuous Representation-based NonLocal method (termed as CRNL), which has two innovative features as compared with classical nonlocal methods. First, based on the continuous representation, our CRNL unifies the measure of self-similarity for on-meshgrid and off-meshgrid data and thus is naturally suitable for both of them. Second, the nonlocal continuous groups can be more compactly and efficiently represented by the coupled low-rank function factorization, which simultaneously exploits the similarity within each group and across different groups, while classical nonlocal methods neglect the similarity across groups. This elaborately designed coupled mechanism allows our method to enjoy favorable performance over conventional NSS methods in terms of both effectiveness and efficiency. Extensive multi-dimensional data processing experiments on-meshgrid (e.g., image inpainting and image denoising) and off-meshgrid (e.g., climate data prediction and point cloud recovery) validate the versatility, effectiveness, and efficiency of our CRNL as compared with state-of-the-art methods.
Paper Structure (28 sections, 3 theorems, 10 equations, 10 figures, 7 tables)

This paper contains 28 sections, 3 theorems, 10 equations, 10 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Nonlocal Self-Similarity-Based Methods
  4. Basis Function-Based Methods
  5. Continuous Representation-Based Methods
  6. The Proposed Method
  7. Preliminaries
  8. Overview of the Proposed CRNL
  9. Detailed Algorithm of CRNL
  10. Similarity within Each Group and across Different Groups
  11. The Design of Regularization Term
  12. Experiments
  13. On Meshgrid Data
  14. Image Inpainting
  15. Image Denoising
  16. ...and 13 more sections

Key Result

Lemma 1

(Tensor Tucker factorization SIAM_review) The Tucker rank of a $N$-th-order tensor ${\mathcal{X}}\in{\mathbb R}^{n_1\times \cdots\times n_N}$ is a vector defined as ${\rm rank}_T({\mathcal{X}}):=({\rm rank}({\bf X}^{(1)}), {\rm rank}({\bf X}^{(2)}),\cdots,{\rm rank}({\bf X}^{(N)}))$.

Figures (10)

  • Figure 1: The overall flowchart of our continuous representation-based nonlocal method for data recovery. Here, we take the image inpainting and point cloud recovery as examlpes.
  • Figure 2: Illustrations of the proposed basic cubes splitting and grouping process in the three-dimensional case ($N=3$) with number of units $n_1=n_2=5$ and cube size $p=2$. Here, $D_t$ denotes the basic continuous cube, $D_l^{\rm key}$ denotes a key continuous cube, and $D_{l_1},D_{l_2}$ denote the two extracted similar continuous cubes of the $l$-th key cube $D_l^{\rm key}$.
  • Figure 3: From upper to lower: The results of image inpainting by different methods on color images Peppers, Plane, Sailboat, and House with sampling rate 0.2.
  • Figure 4: From upper to lower: The results of image inpainting by different methods on MSIs Toys and Cloth with sampling rate 0.1, and videos Foreman and Carphone with sampling rate 0.25.
  • Figure 5: From upper to lower: The results of multi-dimensional image denoising by different methods on MSIs Cups and Fruits with noisy level $\sigma =0.2$.
  • ...and 5 more figures

Theorems & Definitions (3)

  • Lemma 1
  • Lemma 2
  • Lemma 3