Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NeurTV: Total Variation on the Neural Domain

Yisi Luo, Xile Zhao, Kai Ye, Deyu Meng

TL;DR

NeurTV introduces a total-variation regularization defined on a neural, continuous data domain by penalizing the input-derivatives of a deep network that represents the data. This neural-domain TV eliminates discretization error, accommodates both meshgrid and non-meshgrid data, and naturally supports directional and higher-order derivatives through differentiable networks. The authors ground NeurTV in a variational-approximation framework, connect it to classical TV, and develop variants such as arbitrary-resolution and space-variant NeurTV. Extensive experiments across image denoising, inpainting, hyperspectral denoising, point-cloud recovery, and spatial-transcriptomics reconstruction demonstrate consistent performance gains over traditional TV and related baselines, highlighting the method’s flexibility and broad applicability.

Abstract

Recently, we have witnessed the success of total variation (TV) for many imaging applications. However, traditional TV is defined on the original pixel domain, which limits its potential. In this work, we suggest a new TV regularization defined on the neural domain. Concretely, the discrete data is implicitly and continuously represented by a deep neural network (DNN), and we use the derivatives of DNN outputs w.r.t. input coordinates to capture local correlations of data. As compared with classical TV on the original domain, the proposed TV on the neural domain (termed NeurTV) enjoys the following advantages. First, NeurTV is free of discretization error induced by the discrete difference operator. Second, NeurTV is not limited to meshgrid but is suitable for both meshgrid and non-meshgrid data. Third, NeurTV can more exactly capture local correlations across data for any direction and any order of derivatives attributed to the implicit and continuous nature of neural domain. We theoretically reinterpret NeurTV under the variational approximation framework, which allows us to build the connection between NeurTV and classical TV and inspires us to develop variants (e.g., space-variant NeurTV). Extensive numerical experiments with meshgrid data (e.g., color and hyperspectral images) and non-meshgrid data (e.g., point clouds and spatial transcriptomics) showcase the effectiveness of the proposed methods.

NeurTV: Total Variation on the Neural Domain

TL;DR

NeurTV introduces a total-variation regularization defined on a neural, continuous data domain by penalizing the input-derivatives of a deep network that represents the data. This neural-domain TV eliminates discretization error, accommodates both meshgrid and non-meshgrid data, and naturally supports directional and higher-order derivatives through differentiable networks. The authors ground NeurTV in a variational-approximation framework, connect it to classical TV, and develop variants such as arbitrary-resolution and space-variant NeurTV. Extensive experiments across image denoising, inpainting, hyperspectral denoising, point-cloud recovery, and spatial-transcriptomics reconstruction demonstrate consistent performance gains over traditional TV and related baselines, highlighting the method’s flexibility and broad applicability.

Abstract

Recently, we have witnessed the success of total variation (TV) for many imaging applications. However, traditional TV is defined on the original pixel domain, which limits its potential. In this work, we suggest a new TV regularization defined on the neural domain. Concretely, the discrete data is implicitly and continuously represented by a deep neural network (DNN), and we use the derivatives of DNN outputs w.r.t. input coordinates to capture local correlations of data. As compared with classical TV on the original domain, the proposed TV on the neural domain (termed NeurTV) enjoys the following advantages. First, NeurTV is free of discretization error induced by the discrete difference operator. Second, NeurTV is not limited to meshgrid but is suitable for both meshgrid and non-meshgrid data. Third, NeurTV can more exactly capture local correlations across data for any direction and any order of derivatives attributed to the implicit and continuous nature of neural domain. We theoretically reinterpret NeurTV under the variational approximation framework, which allows us to build the connection between NeurTV and classical TV and inspires us to develop variants (e.g., space-variant NeurTV). Extensive numerical experiments with meshgrid data (e.g., color and hyperspectral images) and non-meshgrid data (e.g., point clouds and spatial transcriptomics) showcase the effectiveness of the proposed methods.
Paper Structure (33 sections, 13 theorems, 52 equations, 21 figures, 6 tables, 1 algorithm)

This paper contains 33 sections, 13 theorems, 52 equations, 21 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Interpolation and Rotation-based Methods
  4. Basis Function-based Methods
  5. Deep Learning-based Total Variation Methods
  6. Graph and Mesh-based Methods
  7. The Proposed Methods
  8. Notations
  9. Data Recovery Model Based on Continuous Representation
  10. Continuous Representation-based NeurTV
  11. Extensions of NeurTV
  12. Higher-order NeurTV
  13. Directional NeurTV
  14. Spatial-spectral NeurTV
  15. Justification of NeurTV from Variational Approximation
  16. ...and 18 more sections

Key Result

Lemma 3.3

\newlabellemma_DTV0 Given a function $f({\bf x}):\Omega\rightarrow{\mathbb R}$ that is differentiable w.r.t. the input $\bf x$, where $\Omega \subset{\mathbb R}^2$, the directional derivative along the direction ${\bf e}=(\cos\theta,\sin\theta)$ where $\theta\in[0,2\pi)$ is defined as Such directional derivative of $f(\cdot)$ is equivalent to $\nabla_{\bf e}f({\bf x}) = \frac{\partial f({\bf x})

Figures (21)

  • Figure 1: The histograms of the local differences between adjacent pixels of the image "Peppers". \newlabelfig_hist0
  • Figure 1: Our NeurTV regularization is a basic building block that can be combined with different DNNs $f_\Theta(\cdot)$. For instance we consider (a) the sine function-based DNN sine, (b) the positional encoding (PE)-based DNN NIPS_PE, and (c) the tensor factorization (TF)-based DNN TPAMI_Luo to test the proposed NeurTV method. \newlabelfig_INR0
  • Figure 1: The results of image denoising by different methods on "Peppers", "Boat", and "House" with Gaussian noise under noise deviation 0.1. Here, DIPWTV is a weighted TV-based method with space-variant scales, and HDTV is a higher-order directional TV-based method. \newlabelfig_image_denoising0
  • Figure 2: The results of image denoising on "Peppers" and the results of point cloud recovery on "Mario" by using classical TV and the proposed NeurTV. NeurTV is suitable for both meshgrid and non-meshgrid data (e.g., point cloud), while TV is not suitable for non-meshgrid data. Moreover, NeurTV can better capture directional features by using the directional derivatives of DNN in the continuous domain. \newlabelfig_NeurTV0
  • Figure 2: The results of image inpainting using different DNNs (sine function-based sine, PE-based NIPS_PE, and TF-based TPAMI_Luo DNNs) with and without the NeurTV regularization \ref{['WGV']}. \newlabelfig_inr_img0
  • ...and 16 more figures

Theorems & Definitions (26)

  • Definition 3.1: NeurTV
  • Definition 3.2
  • Lemma 3.3
  • Proof 1
  • Definition 3.4: Directional NeurTV
  • Definition 3.5: Spatial-spectral NeurTV
  • Lemma 3.6: Total variation of function TV
  • Lemma 3.7
  • Proof 2
  • Lemma 3.8: Total variation based on uniform partitions
  • ...and 16 more