Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SDIP: Self-Reinforcement Deep Image Prior Framework for Image Processing

Ziyu Shu, Zhixin Pan

TL;DR

SDIP extends the Deep Image Prior (DIP) by introducing a self-reinforcement mechanism that iteratively updates the DIP input using a steering algorithm guided by the current output. It leverages two key observations: a strong input-output correlation in DIP and a predictable influence of input changes on output, enabling the integration of traditional optimization priors as steering guidance. The framework operates without any training data and demonstrates improved stability and performance across highly ill-posed imaging tasks, including Computed Tomography reconstruction from few views or limited angles, deblurring, and single-image super-resolution. The results show SDIP consistently outperforms original DIP and benefits from incorporating additional priors, while noting remaining sensitivity to noise and outlining future enhancements via more sophisticated steering strategies and alternative network architectures.

Abstract

Deep image prior (DIP) proposed in recent research has revealed the inherent trait of convolutional neural networks (CNN) for capturing substantial low-level image statistics priors. This framework efficiently addresses the inverse problems in image processing and has induced extensive applications in various domains. However, as the whole algorithm is initialized randomly, the DIP algorithm often lacks stability. Thus, this method still has space for further improvement. In this paper, we propose the self-reinforcement deep image prior (SDIP) as an improved version of the original DIP. We observed that the changes in the DIP networks' input and output are highly correlated during each iteration. SDIP efficiently utilizes this trait in a reinforcement learning manner, where the current iteration's output is utilized by a steering algorithm to update the network input for the next iteration, guiding the algorithm toward improved results. Experimental results across multiple applications demonstrate that our proposed SDIP framework offers improvement compared to the original DIP method and other state-of-the-art methods.

SDIP: Self-Reinforcement Deep Image Prior Framework for Image Processing

TL;DR

SDIP extends the Deep Image Prior (DIP) by introducing a self-reinforcement mechanism that iteratively updates the DIP input using a steering algorithm guided by the current output. It leverages two key observations: a strong input-output correlation in DIP and a predictable influence of input changes on output, enabling the integration of traditional optimization priors as steering guidance. The framework operates without any training data and demonstrates improved stability and performance across highly ill-posed imaging tasks, including Computed Tomography reconstruction from few views or limited angles, deblurring, and single-image super-resolution. The results show SDIP consistently outperforms original DIP and benefits from incorporating additional priors, while noting remaining sensitivity to noise and outlining future enhancements via more sophisticated steering strategies and alternative network architectures.

Abstract

Deep image prior (DIP) proposed in recent research has revealed the inherent trait of convolutional neural networks (CNN) for capturing substantial low-level image statistics priors. This framework efficiently addresses the inverse problems in image processing and has induced extensive applications in various domains. However, as the whole algorithm is initialized randomly, the DIP algorithm often lacks stability. Thus, this method still has space for further improvement. In this paper, we propose the self-reinforcement deep image prior (SDIP) as an improved version of the original DIP. We observed that the changes in the DIP networks' input and output are highly correlated during each iteration. SDIP efficiently utilizes this trait in a reinforcement learning manner, where the current iteration's output is utilized by a steering algorithm to update the network input for the next iteration, guiding the algorithm toward improved results. Experimental results across multiple applications demonstrate that our proposed SDIP framework offers improvement compared to the original DIP method and other state-of-the-art methods.
Paper Structure (20 sections, 4 equations, 14 figures, 8 tables, 1 algorithm)

This paper contains 20 sections, 4 equations, 14 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. The Inverse Problems
  4. Deep Image Prior
  5. Proposed Method
  6. Correlation Between the Initial Input and Final Output in DIP
  7. Correlation of Variations between Input and Output
  8. Self-Reinforcement DIP
  9. DIP Network for Image Generation
  10. Steering Algorithm for Introducing Additional Prior
  11. Input Modification with Dynamic Step Size
  12. SDIP Workflow
  13. Applications
  14. Selection of the Hyperparameters
  15. Computed Tomography (CT) Reconstruction
  16. ...and 5 more sections

Figures (14)

  • Figure 1: The reconstruction SNR under different numbers of iterations with noisy inputs.
  • Figure 2: The reconstruction SNR under different numbers of iterations with different inputs.
  • Figure 3: Image inpainting using different inputs. (a) the inpainting image; (b) using uniformly distributed random vector as network input, the corresponding inpainting result is (c); (d) using an image with constant gradient as network input, the corresponding inpainting result is (e).
  • Figure 4: The limited-angle ($0^\circ$ to $120^\circ$$1$ view per degree) CT reconstruction of Forbild phantom. (a) Steepest Descent; (b) CDIP, using (a) as network input; (c) DIP, using Gaussian random vector as network input; (d) the proposed SDIP method.
  • Figure 5: The strong correlation between the change of network input and the corresponding output. The $9$ images on the left are used in the experiments in Section \ref{['sec:IOrelation']}. The images on the right are the mean output changes when the network input is replaced by the fourth, fifth, eighth, and ninth images on the left. The first row corresponds to the DIP method, and the second row corresponds to the CDIP method.
  • ...and 9 more figures