Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Real-World Adverse Weather Image Restoration via Dual-Level Reinforcement Learning with High-Quality Cold Start

Fuyang Liu, Jiaqi Xu, Xiaowei Hu

TL;DR

The paper tackles real-world adverse weather image restoration by addressing the gap between synthetic-data training and real-world performance. It introduces HFLS-Weather, a high-fidelity, depth-consistent dataset of 1 million images synthesized via physics-based weather models, enabling realistic rain, haze, and snow artifacts. A Dual-Level Reinforcement Learning framework combines Perturbation-driven Image Quality Optimization for local refinement of weather-specific restorers with a global meta-controller that dynamically schedules model execution to handle diverse degradations. Experiments on real-world data demonstrate state-of-the-art restoration across snow, haze, and rain, with ablations showing strong benefits from high-quality cold-start pretraining and the synergy between PIQO and multi-agent coordination; the work also discusses societal considerations and the need for safeguards in deployment.

Abstract

Adverse weather severely impairs real-world visual perception, while existing vision models trained on synthetic data with fixed parameters struggle to generalize to complex degradations. To address this, we first construct HFLS-Weather, a physics-driven, high-fidelity dataset that simulates diverse weather phenomena, and then design a dual-level reinforcement learning framework initialized with HFLS-Weather for cold-start training. Within this framework, at the local level, weather-specific restoration models are refined through perturbation-driven image quality optimization, enabling reward-based learning without paired supervision; at the global level, a meta-controller dynamically orchestrates model selection and execution order according to scene degradation. This framework enables continuous adaptation to real-world conditions and achieves state-of-the-art performance across a wide range of adverse weather scenarios. Code is available at https://github.com/xxclfy/AgentRL-Real-Weather

Real-World Adverse Weather Image Restoration via Dual-Level Reinforcement Learning with High-Quality Cold Start

TL;DR

The paper tackles real-world adverse weather image restoration by addressing the gap between synthetic-data training and real-world performance. It introduces HFLS-Weather, a high-fidelity, depth-consistent dataset of 1 million images synthesized via physics-based weather models, enabling realistic rain, haze, and snow artifacts. A Dual-Level Reinforcement Learning framework combines Perturbation-driven Image Quality Optimization for local refinement of weather-specific restorers with a global meta-controller that dynamically schedules model execution to handle diverse degradations. Experiments on real-world data demonstrate state-of-the-art restoration across snow, haze, and rain, with ablations showing strong benefits from high-quality cold-start pretraining and the synergy between PIQO and multi-agent coordination; the work also discusses societal considerations and the need for safeguards in deployment.

Abstract

Adverse weather severely impairs real-world visual perception, while existing vision models trained on synthetic data with fixed parameters struggle to generalize to complex degradations. To address this, we first construct HFLS-Weather, a physics-driven, high-fidelity dataset that simulates diverse weather phenomena, and then design a dual-level reinforcement learning framework initialized with HFLS-Weather for cold-start training. Within this framework, at the local level, weather-specific restoration models are refined through perturbation-driven image quality optimization, enabling reward-based learning without paired supervision; at the global level, a meta-controller dynamically orchestrates model selection and execution order according to scene degradation. This framework enables continuous adaptation to real-world conditions and achieves state-of-the-art performance across a wide range of adverse weather scenarios. Code is available at https://github.com/xxclfy/AgentRL-Real-Weather

Paper Structure

This paper contains 22 sections, 8 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Adverse Weather Image Restoration
  4. Large-Scale, Agent-Based, and Perturbation Methods
  5. High-Fidelity Large-Scale Weather Dataset
  6. Dataset Overview
  7. Weather-Related Artifact Simulation
  8. Dataset Comparison
  9. Dual-Level Reinforcement Learning Framework
  10. Perturbation-driven Image Quality Optimization
  11. Muti-Agent System
  12. Training Strategy
  13. Experimental Results
  14. Performance Comparison with State-of-the-Art Methods
  15. Quantitative comparison.
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Weather-degraded images from Foggy Cityscapes-DBF SDHV18, RESIDE-OTS li2018benchmarking, RainCityscapes cityscapes_rain, and Snow100K liu2018desnownet showcase artifacts such as ghosting and uneven weather effects resulting from depth estimation errors. Note that RESIDE-OTS does not provide public depth maps, and Snow100K lacks depth data. The three rows represent clean images, depth maps, and weather-degraded images, respectively.
  • Figure 2: Architecture of Perturbation-Driven Image Quality Optimization and Multi-Agent System.
  • Figure 3: Visual comparisons of real images under haze, snow, and rain, with chen2022learningguo2024onerestoreluo2024controllingpotlapalli2023promptirzhu2023learning.
  • Figure 4: Visual ablation study of the framework components.