TPGDiff: Hierarchical Triple-Prior Guided Diffusion for Image Restoration
Yanjie Tu, Qingsen Yan, Axi Niu, Jiacong Tang
TL;DR
TPGDiff addresses the challenge of restoring high-quality images from multi-faceted degradations with a unified diffusion-based framework. It introduces a hierarchical triple-prior strategy, applying semantic priors in deep layers via distillation-driven features, structural priors in shallow layers through a multi-cue token aggregator and a Structural Adapter, and degradation priors for stage-aware guidance via a CLIP-based extractor and a degradation-aware time modulation. The diffusion process is governed by a reverse-time SDE conditioned on these priors: $\mathrm{d}\mathbf{x}=\left[\theta_t(\mu-\mathbf{x})-\sigma_t^{2}\, s_{\boldsymbol{\theta}}(\mathbf{x},t;\boldsymbol{\mu},\mathbf{z}_{\mathrm{sem}},\mathbf{z}_{\mathrm{struct}},\mathbf{z}_{\mathrm{deg}})\right]\mathrm{d}t + \sigma_t\,\mathrm{d}\hat{\mathbf{w}}$, enabling joint exploitation of semantic, structural, and degradation cues throughout restoration. The semantic priors are obtained via a teacher–student distillation framework and are integrated using cross-attention in deep layers; structural priors combine depth, segmentation, and DoG cues into compact tokens that guide shallow layers; degradation priors are learned during training and modulate the temporal conditioning of the diffusion. Experiments across nine degradation types and multi-task settings demonstrate that TPGDiff achieves state-of-the-art performance and strong generalization, offering a practical all-in-one solution for diverse restoration scenarios.
Abstract
All-in-one image restoration aims to address diverse degradation types using a single unified model. Existing methods typically rely on degradation priors to guide restoration, yet often struggle to reconstruct content in severely degraded regions. Although recent works leverage semantic information to facilitate content generation, integrating it into the shallow layers of diffusion models often disrupts spatial structures (\emph{e.g.}, blurring artifacts). To address this issue, we propose a Triple-Prior Guided Diffusion (TPGDiff) network for unified image restoration. TPGDiff incorporates degradation priors throughout the diffusion trajectory, while introducing structural priors into shallow layers and semantic priors into deep layers, enabling hierarchical and complementary prior guidance for image reconstruction. Specifically, we leverage multi-source structural cues as structural priors to capture fine-grained details and guide shallow layers representations. To complement this design, we further develop a distillation-driven semantic extractor that yields robust semantic priors, ensuring reliable high-level guidance at deep layers even under severe degradations. Furthermore, a degradation extractor is employed to learn degradation-aware priors, enabling stage-adaptive control of the diffusion process across all timesteps. Extensive experiments on both single- and multi-degradation benchmarks demonstrate that TPGDiff achieves superior performance and generalization across diverse restoration scenarios. Our project page is: https://leoyjtu.github.io/tpgdiff-project.