Neural Discrimination-Prompted Transformers for Efficient UHD Image Restoration and Enhancement

TL;DR

A super-resolution-guided reconstruction approach, which is guided by super-resolving low-resolution features to facilitate final UHD image restoration and enhancement and enhances the quality of restored images.

Abstract

We propose a simple yet effective UHDPromer, a neural discrimination-prompted Transformer, for Ultra-High-Definition (UHD) image restoration and enhancement. Our UHDPromer is inspired by an interesting observation that there implicitly exist neural differences between high-resolution and low-resolution features, and exploring such differences can facilitate low-resolution feature representation. To this end, we first introduce Neural Discrimination Priors (NDP) to measure the differences and then integrate NDP into the proposed Neural Discrimination-Prompted Attention (NDPA) and Neural Discrimination-Prompted Network (NDPN). The proposed NDPA re-formulates the attention by incorporating NDP to globally perceive useful discrimination information, while the NDPN explores a continuous gating mechanism guided by NDP to selectively permit the passage of beneficial content. To enhance the quality of restored images, we propose a super-resolution-guided reconstruction approach, which is guided by super-resolving low-resolution features to facilitate final UHD image restoration. Experiments show that UHDPromer achieves the best computational efficiency while still maintaining state-of-the-art performance on $3$ UHD image restoration and enhancement tasks, including low-light image enhancement, image dehazing, and image deblurring. The source codes and pre-trained models will be made available at https://github.com/supersupercong/uhdpromer.

Figures (14)

• Figure 1: Overview of our proposed Neural Discrimination-Prompted Transformers (UHDPromer). Our UHDPromer contains $4$ parts: (a) High-Resolution Feature Representation (HRFR); (b) Neural Discrimination-Prompted Transformers (NDPT); (c) Feature Super-Resolution (FeaSR); and (d) SR-Guided Reconstruction (SRG-Recon). Firstly, HRFR explores hierarchical multi-scale high-resolution features, which participate in forming neural discrimination priors (NDP) with low-resolution features. Then, NDPT, which is guided by NDP, learns low-resolution features. Next, FeaSR super-resolves the output features of NDPT. Finally, SRG-Recon reconstructs final images guided by learned SR features in FeaSR.
• Figure 2: Illustration of the Neural Discrimination Priors (NDP) in the different layers by \ref{['eq: NDP']}. We note that the low-resolution features exist obvious content differences compared with high-resolution ones. Specifically, low-resolution features always hand down extensive noises while some structures are blurry. Our NDP can better identify these regions, which can effectively measure whether the structures lie in shallower layers or deeper layers, serving as the discrimination function to help learn more representative features in the low-resolution domain.
• Figure 3: (a) Neural Discrimination-Prompted Attention (NDPA) and (b) Neural Discrimination-Prompted Network (NDPN). The NDPA first performs cross attention between NDP and query vector generated by low-resolution features to globally perceive the NDP, which would be re-performed by the attention computation between rest key and value vectors of low-resolution features to further explore useful learned information. The NDPN considers the continuous gating mechanism guided by the NDP to allow useful information to be passed to facilitate final image reconstruction.
• Figure 4: UHD low-light enhancement on UHD-LLLi2023ICLR_uhdfour under Setting 1. UHDPromer is able to produce clearer results with vivid colors.
• Figure 5: UHD low-light enhancement on UHD-LLLi2023ICLR_uhdfour under Setting 2. UHDPromer is capable of generating clearer results with more natural colors.