Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FourierMamba: Fourier Learning Integration with State Space Models for Image Deraining

Dong Li, Yidi Liu, Xueyang Fu, Senyan Xu, Zheng-Jun Zha

TL;DR

FourierMamba targets image deraining by fusing Fourier-domain priors with Mamba-based correlation across frequencies. The method introduces zigzag-based frequency scanning in the spatial domain and employs Mamba in the channel domain to extract cross-frequency dependencies, implemented within a multiscale FRSSB framework. The approach yields state-of-the-art PSNR/SSIM on multiple rain datasets and is supported by extensive ablations demonstrating the critical role of Fourier priors and the proposed scanning strategies. This work provides a principled way to exploit frequency information for robust rain removal with efficient global modeling.

Abstract

Image deraining aims to remove rain streaks from rainy images and restore clear backgrounds. Currently, some research that employs the Fourier transform has proved to be effective for image deraining, due to it acting as an effective frequency prior for capturing rain streaks. However, despite there exists dependency of low frequency and high frequency in images, these Fourier-based methods rarely exploit the correlation of different frequencies for conjuncting their learning procedures, limiting the full utilization of frequency information for image deraining. Alternatively, the recently emerged Mamba technique depicts its effectiveness and efficiency for modeling correlation in various domains (e.g., spatial, temporal), and we argue that introducing Mamba into its unexplored Fourier spaces to correlate different frequencies would help improve image deraining. This motivates us to propose a new framework termed FourierMamba, which performs image deraining with Mamba in the Fourier space. Owning to the unique arrangement of frequency orders in Fourier space, the core of FourierMamba lies in the scanning encoding of different frequencies, where the low-high frequency order formats exhibit differently in the spatial dimension (unarranged in axis) and channel dimension (arranged in axis). Therefore, we design FourierMamba that correlates Fourier space information in the spatial and channel dimensions with distinct designs. Specifically, in the spatial dimension Fourier space, we introduce the zigzag coding to scan the frequencies to rearrange the orders from low to high frequencies, thereby orderly correlating the connections between frequencies; in the channel dimension Fourier space with arranged orders of frequencies in axis, we can directly use Mamba to perform frequency correlation and improve the channel information representation.

FourierMamba: Fourier Learning Integration with State Space Models for Image Deraining

TL;DR

FourierMamba targets image deraining by fusing Fourier-domain priors with Mamba-based correlation across frequencies. The method introduces zigzag-based frequency scanning in the spatial domain and employs Mamba in the channel domain to extract cross-frequency dependencies, implemented within a multiscale FRSSB framework. The approach yields state-of-the-art PSNR/SSIM on multiple rain datasets and is supported by extensive ablations demonstrating the critical role of Fourier priors and the proposed scanning strategies. This work provides a principled way to exploit frequency information for robust rain removal with efficient global modeling.

Abstract

Image deraining aims to remove rain streaks from rainy images and restore clear backgrounds. Currently, some research that employs the Fourier transform has proved to be effective for image deraining, due to it acting as an effective frequency prior for capturing rain streaks. However, despite there exists dependency of low frequency and high frequency in images, these Fourier-based methods rarely exploit the correlation of different frequencies for conjuncting their learning procedures, limiting the full utilization of frequency information for image deraining. Alternatively, the recently emerged Mamba technique depicts its effectiveness and efficiency for modeling correlation in various domains (e.g., spatial, temporal), and we argue that introducing Mamba into its unexplored Fourier spaces to correlate different frequencies would help improve image deraining. This motivates us to propose a new framework termed FourierMamba, which performs image deraining with Mamba in the Fourier space. Owning to the unique arrangement of frequency orders in Fourier space, the core of FourierMamba lies in the scanning encoding of different frequencies, where the low-high frequency order formats exhibit differently in the spatial dimension (unarranged in axis) and channel dimension (arranged in axis). Therefore, we design FourierMamba that correlates Fourier space information in the spatial and channel dimensions with distinct designs. Specifically, in the spatial dimension Fourier space, we introduce the zigzag coding to scan the frequencies to rearrange the orders from low to high frequencies, thereby orderly correlating the connections between frequencies; in the channel dimension Fourier space with arranged orders of frequencies in axis, we can directly use Mamba to perform frequency correlation and improve the channel information representation.
Paper Structure (14 sections, 13 equations, 12 figures, 3 tables)

This paper contains 14 sections, 13 equations, 12 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminary
  4. Scanning in Fourier Space
  5. FourierMamba
  6. Overall Framework
  7. Fourier Spatial Interaction SSM
  8. Fourier Channel Evolution SSM
  9. Optimization
  10. Experiment
  11. Experimental Settings
  12. Comparison with State-of-the-art Methods
  13. Ablation Studies
  14. Conclusion

Figures (12)

  • Figure 1: Observation and comparison of different frequency modeling methods. (a) Observation of the amplitude spectrum exchange. The degradation is mainly in amplitude components, so the Fourier transform helps to disentangle the image content and rain. (b) The commonly used $1\times1$ convolution cannot model the relationship between different frequencies. (c) Previous scanning in Fourier space will fail to establish the ordered dependence between frequencies. (d) Our proposed method achieves ordered frequency dependence from low to high (or vice versa), thus fully utilizing frequency information.
  • Figure 2: Our proposed Fourier space scanning method in the spatial dimension (top) and channel dimension (bottom). For simplicity, only one direction is shown for each scanning method, and in fact each method also performs a scan opposite to that shown.
  • Figure 3: The overall architecture of the FourierMamba. Our FourierMamba consists of multiscale hierarchical design Fourier Residual State-Space Blocks(FRSSB). The core modules of FRSSB are Fourier Spatial Interaction SSM(FSI-SSM) and Fourier Channel Evolution SSM(FCS-SSM).
  • Figure 4: The error map between the GT and the restored images using various scanning methods in Fourier space. The two scanning methods we propose can achieve smaller errors than using classical scanning method vmamba. And the combination of the two scanning methods is better than either one.
  • Figure 5: Visual quality comparison on Rain100H yang2017deep. Zoom in for better visualization.
  • ...and 7 more figures