Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Prototype Unit for Image De-raining using Time-Lapse Data

Jaehoon Cho, Minjung Yoo, Jini Yang, Sunok Kim

TL;DR

This work tackles single-image de-raining under realistic constraints by introducing the Rain-streak Prototype Unit (RsPU), a memory-efficient attention-based mechanism that encodes rain-streak features as real-time prototypes derived from time-lapse data. By integrating RsPU into an encoder–decoder framework and applying a comprehensive loss suite—including a novel feature prototype loss that enforces cohesion and diversity among prototypes—the method captures diverse rain-streak patterns while preserving background content. The approach demonstrates competitive performance on real and synthetic rain benchmarks, while significantly reducing memory usage and enabling deployment on devices with limited resources. The results suggest strong generalization to real rain and hazy conditions, highlighting practical implications for robust outdoor vision tasks.

Abstract

We address the challenge of single-image de-raining, a task that involves recovering rain-free background information from a single rain image. While recent advancements have utilized real-world time-lapse data for training, enabling the estimation of consistent backgrounds and realistic rain streaks, these methods often suffer from computational and memory consumption, limiting their applicability in real-world scenarios. In this paper, we introduce a novel solution: the Rain Streak Prototype Unit (RsPU). The RsPU efficiently encodes rain streak-relevant features as real-time prototypes derived from time-lapse data, eliminating the need for excessive memory resources. Our de-raining network combines encoder-decoder networks with the RsPU, allowing us to learn and encapsulate diverse rain streak-relevant features as concise prototypes, employing an attention-based approach. To ensure the effectiveness of our approach, we propose a feature prototype loss encompassing cohesion and divergence components. This loss function captures both the compactness and diversity aspects of the prototypical rain streak features within the RsPU. Our method evaluates various de-raining benchmarks, accompanied by comprehensive ablation studies. We show that it can achieve competitive results in various rain images compared to state-of-the-art methods.

A Prototype Unit for Image De-raining using Time-Lapse Data

TL;DR

This work tackles single-image de-raining under realistic constraints by introducing the Rain-streak Prototype Unit (RsPU), a memory-efficient attention-based mechanism that encodes rain-streak features as real-time prototypes derived from time-lapse data. By integrating RsPU into an encoder–decoder framework and applying a comprehensive loss suite—including a novel feature prototype loss that enforces cohesion and diversity among prototypes—the method captures diverse rain-streak patterns while preserving background content. The approach demonstrates competitive performance on real and synthetic rain benchmarks, while significantly reducing memory usage and enabling deployment on devices with limited resources. The results suggest strong generalization to real rain and hazy conditions, highlighting practical implications for robust outdoor vision tasks.

Abstract

We address the challenge of single-image de-raining, a task that involves recovering rain-free background information from a single rain image. While recent advancements have utilized real-world time-lapse data for training, enabling the estimation of consistent backgrounds and realistic rain streaks, these methods often suffer from computational and memory consumption, limiting their applicability in real-world scenarios. In this paper, we introduce a novel solution: the Rain Streak Prototype Unit (RsPU). The RsPU efficiently encodes rain streak-relevant features as real-time prototypes derived from time-lapse data, eliminating the need for excessive memory resources. Our de-raining network combines encoder-decoder networks with the RsPU, allowing us to learn and encapsulate diverse rain streak-relevant features as concise prototypes, employing an attention-based approach. To ensure the effectiveness of our approach, we propose a feature prototype loss encompassing cohesion and divergence components. This loss function captures both the compactness and diversity aspects of the prototypical rain streak features within the RsPU. Our method evaluates various de-raining benchmarks, accompanied by comprehensive ablation studies. We show that it can achieve competitive results in various rain images compared to state-of-the-art methods.
Paper Structure (26 sections, 10 equations, 6 figures, 3 tables)

This paper contains 26 sections, 10 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Single Image De-raining
  4. Attention mechanism
  5. Proposed Method
  6. Rain-streak Prototype Unit (RsPU)
  7. Loss Functions
  8. Feature Prototype Loss
  9. Background Consistency Loss
  10. Cross Consistency Loss
  11. Self Consistency Loss
  12. Total Loss
  13. Network Architectures
  14. Experiment
  15. Datasets
  16. ...and 11 more sections

Figures (6)

  • Figure 1: The de-rained results using the state-of-the-art de-rained method and the proposed method on a real rainy image. Our method generates better-de-raining results than state-of-the-art methods on the real rainy image.
  • Figure 2: The overall framework of our method.
  • Figure 3: Qualitative results on real rain images. From left to right, Input image, JORDER-E, MPRNet, RCDNet, MemoryNet, MAXIM, Restormer and Ours.
  • Figure 4: Qualitative results on RealDataset wang2019spatial. (a) Input, (b) MemoryNet, (c) JORDER-E, (d) RCDNet, (e) Restormer, (f) MAXIM, (g) DRSformer, and (h) ours.
  • Figure 5: Visualization results of de-raining results, in terms of the loss functions.
  • ...and 1 more figures