Vision-Language Controlled Deep Unfolding for Joint Medical Image Restoration and Segmentation

TL;DR

This work tackles the All-in-One Medical Image Restoration and Segmentation (AiOMIRS) problem by proposing VL-DUN, a unified framework that jointly restores HQ medical images and segments anatomy. It combines a Vision-Language Prior Extraction module, using a fine-tuned CLIP to produce Modality and Degradation Priors, with a Frequency-Aware Mamba-based Deep Unfolding Network that unfolds a PGD-like optimization in $K$ stages. The approach explicitly addresses distribution shifts and spectral bias, achieving state-of-the-art results (e.g., +0.92 dB PSNR, +9.76% Dice) on multi-modal benchmarks, and demonstrations of strong AiOMedIS performance. The findings support a synergistic view where restoration and segmentation mutually regularize and refine each other, enabling more robust clinical pipelines.

Abstract

We propose VL-DUN, a principled framework for joint All-in-One Medical Image Restoration and Segmentation (AiOMIRS) that bridges the gap between low-level signal recovery and high-level semantic understanding. While standard pipelines treat these tasks in isolation, our core insight is that they are fundamentally synergistic: restoration provides clean anatomical structures to improve segmentation, while semantic priors regularize the restoration process. VL-DUN resolves the sub-optimality of sequential processing through two primary innovations. (1) We formulate AiOMIRS as a unified optimization problem, deriving an interpretable joint unfolding mechanism where restoration and segmentation are mathematically coupled for mutual refinement. (2) We introduce a frequency-aware Mamba mechanism to capture long-range dependencies for global segmentation while preserving the high-frequency textures necessary for restoration. This allows for efficient global context modeling with linear complexity, effectively mitigating the spectral bias of standard architectures. As a pioneering work in the AiOMIRS task, VL-DUN establishes a new state-of-the-art across multi-modal benchmarks, improving PSNR by 0.92 dB and the Dice coefficient by 9.76\%. Our results demonstrate that joint collaborative learning offers a superior, more robust solution for complex clinical workflows compared to isolated task processing. The codes are provided in https://github.com/cipi666/VLDUN.

Figures (14)

• Figure 1: Our VL-DUN leverages CLIP to extract explicit dual priors regarding modality and degradation. Our DUN uses frequency-decoupling strategy, the low-frequency feature is processed by GDFN, while Mamba captures high-frequency texture.
• Figure 2: The overall architecture of the proposed VL-DUN. Our VL-DUN comprises two main components: (1) The Vision-Language Prior Extraction Module (Top): Given a LQ input, it utilizes a fine-tuned CLIP model to extract explicit semantic cues. (2) The Attention-Mamba based Deep Unfolding Network (Bottom): Mathematically grounded in the Proximal Gradient Descent algorithm, the network unfolds the optimization into $K$ stages arranged in a hierarchical architecture.
• Figure 3: By comparison, our fine-tune CLIP can distinguish 8 modalities much better than basic CLIP.
• Figure 4: Although the original CLIP has high performance in natural image degradation recognition, it lacks generality in the medical image degradation recognition scene. Our fine-tuned CLIP is able to recognize different degradation well.
• Figure 5: Spectral analysis reveals that standard Mamba exhibits inherent low-pass filtering properties, suppressing high-frequency textures alongside noise. In contrast, our frequency-decoupled strategy enables it to effectively distinguish anatomical details from noise by leveraging global context on the high-frequency band.