A Two-Stage Adverse Weather Semantic Segmentation Method for WeatherProof Challenge CVPR 2024 Workshop UG2+

TL;DR

This work tackles semantic segmentation under adverse weather by introducing a two-stage framework that first uses a low-rank tensor-based deraining method to generate high-quality pseudo ground truths from multi-frame video sequences, then trains a CNN-based segmentation model (InternImage) on these pseudo labels. The LLRT stage exploits temporal and non-local self-similarity information to suppress rain and fog artifacts, producing pseudo labels that align well with real scene content. The approach achieves $0.43$ in $mIoU$ and ranks 4th on the WeatherProof benchmark, demonstrating robustness to degraded imagery and potential applicability to other robustness-critical perception tasks. Overall, the method leverages temporal information to enhance supervision for semantic segmentation in adverse conditions and offers a practical strategy for robust real-world perception systems.

Abstract

This technical report presents our team's solution for the WeatherProof Dataset Challenge: Semantic Segmentation in Adverse Weather at CVPR'24 UG2+. We propose a two-stage deep learning framework for this task. In the first stage, we preprocess the provided dataset by concatenating images into video sequences. Subsequently, we leverage a low-rank video deraining method to generate high-fidelity pseudo ground truths. These pseudo ground truths offer superior alignment compared to the original ground truths, facilitating model convergence during training. In the second stage, we employ the InternImage network to train for the semantic segmentation task using the generated pseudo ground truths. Notably, our meticulously designed framework demonstrates robustness to degraded data captured under adverse weather conditions. In the challenge, our solution achieved a competitive score of 0.43 on the Mean Intersection over Union (mIoU) metric, securing a respectable rank of 4th.