Joint Conditional Diffusion Model for Image Restoration with Mixed Degradations

TL;DR

This work tackles image restoration under mixed weather degradations by combining a physics-grounded mixed degradation model with a Joint Conditional Diffusion Model (JCDM) that uses both the degraded image and a degradation mask as guidance. A refinement stage incorporating an Uncertainty Estimation Block (UEB) further enhances color and detail by modeling aleatoric and epistemic uncertainty and modulating features accordingly. The model is trained with a tailored loss that blends reconstruction objectives and uncertainty-aware terms, and it demonstrates superior performance on multi-weather and task-specific benchmarks compared to state-of-the-art baselines, while achieving substantially faster inference than diffusion-only methods. Overall, the approach provides a robust, efficient framework for blind image restoration in scenarios with complex, mixed degradations and has practical implications for surveillance, autonomous driving, and outdoor vision systems under adverse conditions.

Abstract

Image restoration is rather challenging in adverse weather conditions, especially when multiple degradations occur simultaneously. Blind image decomposition was proposed to tackle this issue, however, its effectiveness heavily relies on the accurate estimation of each component. Although diffusion-based models exhibit strong generative abilities in image restoration tasks, they may generate irrelevant contents when the degraded images are severely corrupted. To address these issues, we leverage physical constraints to guide the whole restoration process, where a mixed degradation model based on atmosphere scattering model is constructed. Then we formulate our Joint Conditional Diffusion Model (JCDM) by incorporating the degraded image and degradation mask to provide precise guidance. To achieve better color and detail recovery results, we further integrate a refinement network to reconstruct the restored image, where Uncertainty Estimation Block (UEB) is employed to enhance the features. Extensive experiments performed on both multi-weather and weather-specific datasets demonstrate the superiority of our method over state-of-the-art competing methods.

Figures (9)

• Figure 1: Comparative results of image restoration techniques under the mixed degradations (rain streak + heavy haze). Neither WeatherDiff ozdenizci2023restoring nor BIDeN han2022blind approaches restore the sky area effectively. The error map highlight the effectiveness of our method in addressing this complex challenge.
• Figure 2: Overall architecture of the proposed algorithm. The pipeline is as follows. Firstly, we formulate the degraded image with random mixed degradations using our constructed model (equation (8)). Subsequently, the mask prediction branch is leveraged to estimate the degradation mask corresponding to the degraded image. This predicted mask, along with the degraded image, serves as conditions of the diffusion model. The initial restoration results obtained from the stage are then fed into the refinement network. Finally, the restored image is obtained.
• Figure 3: The overall Uncertainty Estimation Block (UEB) structure, which conducts aleatoric and epistemic uncertainty modeling through two separate branches, respectively.
• Figure 4: Comparison of the restoration results with and without refinement.
• Figure 5: Qualitative results of joint degradation removal under 6 cases. Some areas are highlighted in colored rectangles for a better visualization and comparison.