Analysis of Deep Image Prior and Exploiting Self-Guidance for Image Reconstruction

TL;DR

This work analyzes why Deep Image Prior (DIP) can reconstruct images from undersampled measurements and how training dynamics in the neural tangent kernel (NTK) regime interact with forward operators such as those in MRI. It derives conditions under which DIP can overfit or fail to recover high-frequency content, and proposes a self-guided DIP that jointly optimizes network weights and input with a denoiser-based regularizer, removing the need for training data or reference images. The authors demonstrate that self-guided DIP outperforms vanilla DIP, reference-guided DIP, and several supervised baselines on MRI reconstruction tasks (fastMRI knee/brain, Stanford FSE) and on image inpainting (CBSD68), with reduced spectral bias and negligible overfitting. This unsupervised, instance-adaptive approach yields competitive or superior results across diverse datasets while maintaining data-consistency and robustness to distribution shifts, indicating strong practical potential for medical imaging and related inverse problems.

Abstract

The ability of deep image prior (DIP) to recover high-quality images from incomplete or corrupted measurements has made it popular in inverse problems in image restoration and medical imaging including magnetic resonance imaging (MRI). However, conventional DIP suffers from severe overfitting and spectral bias effects. In this work, we first provide an analysis of how DIP recovers information from undersampled imaging measurements by analyzing the training dynamics of the underlying networks in the kernel regime for different architectures. This study sheds light on important underlying properties for DIP-based recovery. Current research suggests that incorporating a reference image as network input can enhance DIP's performance in image reconstruction compared to using random inputs. However, obtaining suitable reference images requires supervision, and raises practical difficulties. In an attempt to overcome this obstacle, we further introduce a self-driven reconstruction process that concurrently optimizes both the network weights and the input while eliminating the need for training data. Our method incorporates a novel denoiser regularization term which enables robust and stable joint estimation of both the network input and reconstructed image. We demonstrate that our self-guided method surpasses both the original DIP and modern supervised methods in terms of MR image reconstruction performance and outperforms previous DIP-based schemes for image inpainting.

Figures (16)

• Figure 1: Top row: The performance of reconstructing a 1D square signal after applying the Fourier transform using both the Deep Decoder (top middle) and the WCNN (top right) where the y-axis is the magnitude and x-axis is the length. Bottom row left: The RMSE performance for the 1D reconstruction is depicted on the left side. Bottom row right: The two figures at the bottom display the Fourier transform of the left eigenvector matrix for both the Deep Decoder (bottom left) and the WCNN cases (bottom right).
• Figure 2: Top row: the three masks used to compute the frequency band-based metric. Bottom row: reconstruction PSNR plot on the left illustrates the overfitting issue that occurs during MRI reconstruction. Spectral bias also affects the performance of DIP for MRI reconstruction (right plot), as different frequency bands are reconstructed at different rates.
• Figure 3: Self-guided deep image prior: effect of regularization.
• Figure 4: Evolution of the network input in self-guided DIP during training for MRI reconstruction at 4x undersampling. As the loss from \ref{['eq:self_guided_dip']} diminishes, the self-guided input supplies additional data, enabling the neural network to enhance its reconstruction capabilities.
• Figure 5: Flow chart of the proposed self-guided DIP algorithm.

Theorems & Definitions (3)

• Theorem 1
• Theorem 2
• Corollary 1