Bidirectional Multi-Scale Implicit Neural Representations for Image Deraining

TL;DR

The paper addresses image deraining under spatially varying rain by introducing NeRD-Rain, a bidirectional, multi-scale Transformer that incorporates intra-scale implicit neural representations to learn common rain degradation. An inter-scale bidirectional branch (BFPU) enables coarse-to-fine and fine-to-coarse information exchange, forming a closed-loop architecture that enhances cross-scale collaboration. NeRD-Rain demonstrates state-of-the-art performance on synthetic benchmarks and real-world datasets, with a lighter variant (NeRD-Rain-S) offering reduced computational cost. The work contributes a robust, scale-aware framework that improves rain removal under complex, real-world conditions and suggests avenues for extending INR integration to other vision tasks.

Abstract

How to effectively explore multi-scale representations of rain streaks is important for image deraining. In contrast to existing Transformer-based methods that depend mostly on single-scale rain appearance, we develop an end-to-end multi-scale Transformer that leverages the potentially useful features in various scales to facilitate high-quality image reconstruction. To better explore the common degradation representations from spatially-varying rain streaks, we incorporate intra-scale implicit neural representations based on pixel coordinates with the degraded inputs in a closed-loop design, enabling the learned features to facilitate rain removal and improve the robustness of the model in complex scenarios. To ensure richer collaborative representation from different scales, we embed a simple yet effective inter-scale bidirectional feedback operation into our multi-scale Transformer by performing coarse-to-fine and fine-to-coarse information communication. Extensive experiments demonstrate that our approach, named as NeRD-Rain, performs favorably against the state-of-the-art ones on both synthetic and real-world benchmark datasets. The source code and trained models are available at https://github.com/cschenxiang/NeRD-Rain.

Figures (9)

• Figure 1: Illustration of the proposed approach and the currently existing multi-scale solutions. (a) coarse-to-fine scheme zhang2018densityjiang2020multi; (b) multi-patch scheme zamir2021multi; (c) our method. Compared to previous approaches, the method one integrates implicit neural representations (INR) into our bidirectional multi-scale model to form a closed-loop framework, which allows for better exploration of multi-scale information and modeling of complex rain streaks.
• Figure 2: Overall architecture of the proposed bidirectional multi-scale Transformer with implicit neural representations (NeRD-Rain), which consists of intra-scale flows (i.e., INR branch and unequal Transformer branch) and inter-scale flows (i.e., coarse-to-fine and fine-to-coarse bidirectional branches). The proposed INR branch consists of two coordinated-based MLPs with coarse and fine feature grids. We construct an intra-scale shared encoder in the Transformer branch and INR branch, where two types of representation (i.e., scale-specific and common rain ones) are able to complement each other. We formulate all the branches to form a closed-loop network architecture.
• Figure 3: Derained results on the Rain200H dataset yang2017deep. Compared with the derained results in (c)-(k), our method recovers a high-quality image with clearer details. Zooming in the figures offers a better view at the deraining capability.
• Figure 4: Derained results on the SPA-Data wang2019spatial dataset. Compared with the derained results in (c)-(f), our method recovers clearer images.
• Figure 5: Derained results on a real-world rainy image from chen2023towards. Compared with the derained results in (b)-(f), our method removes most rain streaks and recovers a clearer image. Zooming in the figures offers a better view at the deraining capability.