Learning Domain-Aware Task Prompt Representations for Multi-Domain All-in-One Image Restoration

TL;DR

This work proposes the first multi-domain all-in-one image restoration method, DATPRL-IR, based on the proposed Domain-Aware Task Prompt Representation Learning, which significantly outperforms existing SOTA image restoration methods, while exhibiting strong generalization capabilities.

Abstract

Recently, significant breakthroughs have been made in all-in-one image restoration (AiOIR), which can handle multiple restoration tasks with a single model. However, existing methods typically focus on a specific image domain, such as natural scene, medical imaging, or remote sensing. In this work, we aim to extend AiOIR to multiple domains and propose the first multi-domain all-in-one image restoration method, DATPRL-IR, based on our proposed Domain-Aware Task Prompt Representation Learning. Specifically, we first construct a task prompt pool containing multiple task prompts, in which task-related knowledge is implicitly encoded. For each input image, the model adaptively selects the most relevant task prompts and composes them into an instance-level task representation via a prompt composition mechanism (PCM). Furthermore, to endow the model with domain awareness, we introduce another domain prompt pool and distill domain priors from multimodal large language models into the domain prompts. PCM is utilized to combine the adaptively selected domain prompts into a domain representation for each input image. Finally, the two representations are fused to form a domain-aware task prompt representation which can make full use of both specific and shared knowledge across tasks and domains to guide the subsequent restoration process. Extensive experiments demonstrate that our DATPRL-IR significantly outperforms existing SOTA image restoration methods, while exhibiting strong generalization capabilities. Code is available at https://github.com/GuangluDong0728/DATPRL-IR.

Figures (13)

• Figure 1: This paper makes a preliminary exploration of multi-domain all-in-one image restoration (MD-AiOIR), aiming at further extending the restoration capability of a single model to a broader range of tasks and image domains, including natural scene, medical imaging, and remote sensing.
• Figure 2: Framework of the proposed DATPRL-IR for multi-domain all-in-one image restoration. By introducing domain-aware task representation learning, DATPRL-IR can fully utilize both specific and shared knowledge across tasks and domains, effectively reducing the learning difficulty of the model and improving its performance.
• Figure 3: A partial visualization of the word clouds generated from the text descriptions produced by LLAVA, and the t-SNE clustering analysis of the text descriptions corresponding to the 9 datasets from different domains and tasks. It can be observed that images from different domains exhibit their own characteristics while also sharing certain overlapping features.
• Figure 4: Comparison of our DATPRL-IR with other SOTA methods on 6-task and 3-domain setting.
• Figure 5: Comparison of our DATPRL-IR with other SOTA methods on 9-task and 3-domain setting.