Modeling Weather Uncertainty for Multi-weather Co-Presence Estimation

TL;DR

Outdoor weather is treated as a continuous, probabilistic phenomenon rather than discrete labels. The authors propose an uncertainty-aware framework (MeFormer) that uses a Gaussian-mixture representation of weather, coupled with prior-posterior learning to capture weather uncertainty, and introduce the MePe dataset with 14 weather categories and probability ground truth. MeFormer achieves state-of-the-art performance on multi-weather co-presence estimation and strong results on related tasks, including adverse-weather semantic segmentation, demonstrating the practical benefits of modeling weather uncertainty. The work provides a physics-grounded, scalable benchmark for robust outdoor perception and sets a foundation for further exploration of continuous weather modeling in vision systems.

Abstract

Images from outdoor scenes may be taken under various weather conditions. It is well studied that weather impacts the performance of computer vision algorithms and needs to be handled properly. However, existing algorithms model weather condition as a discrete status and estimate it using multi-label classification. The fact is that, physically, specifically in meteorology, weather are modeled as a continuous and transitional status. Instead of directly implementing hard classification as existing multi-weather classification methods do, we consider the physical formulation of multi-weather conditions and model the impact of physical-related parameter on learning from the image appearance. In this paper, we start with solid revisit of the physics definition of weather and how it can be described as a continuous machine learning and computer vision task. Namely, we propose to model the weather uncertainty, where the level of probability and co-existence of multiple weather conditions are both considered. A Gaussian mixture model is used to encapsulate the weather uncertainty and a uncertainty-aware multi-weather learning scheme is proposed based on prior-posterior learning. A novel multi-weather co-presence estimation transformer (MeFormer) is proposed. In addition, a new multi-weather co-presence estimation (MePe) dataset, along with 14 fine-grained weather categories and 16,078 samples, is proposed to benchmark both conventional multi-label weather classification task and multi-weather co-presence estimation task. Large scale experiments show that the proposed method achieves state-of-the-art performance and substantial generalization capabilities on both the conventional multi-label weather classification task and the proposed multi-weather co-presence estimation task. Besides, modeling weather uncertainty also benefits adverse-weather semantic segmentation.

Figures (10)

• Figure 1: Discrete labels are insufficient to describe weather because it is transitional. An outdoor scene can observe the co-existence of different weather conditions under a variety of probability levels. First row: sunny and rainy; Second row: foggy and rainy.
• Figure 2: Given a binary sunny-rainy example, feature space illustration on: (a) conventional deterministic pipelines; (b) our probabilistic pipeline with uncertainty quantification. The probabilistic feature space is more capable to describe the scenario where multi-weather conditions co-exist.
• Figure 3: Illustration of multi-weather co-presence estimation by (a) existing deep learning paradigm in a deterministic manner; (b) the proposed paradigm with weather uncertainty. $\mathbf{X}$, $p$, $h$ denote the feature representation, probability prediction and prediction head, respectively.
• Figure 4: The proposed multi-weather co-presence estimation pipeline by modeling the uncertainty of weather features. Black arrows exist in both training & inference stage, while blue arrows only exist in training stage. $\mathbf{I}$, $\mathbf{X}$, $p$, $\hat{p}$ denote the image appearance, feature representation, probability prediction and ground truth, respectively.
• Figure 5: Illustration of the proposed Multi-weather Co-presence Estimation Transformer (MeFormer). It consists of three key components, namely, multi-weather feature embedding (MFE) (Sec. \ref{['sec4.2']}), prior-posterior uncertainty learning (PUL) (Sec. \ref{['sec4.3']}), and multi-weather probability prediction (MPP) (Sec. \ref{['sec4.4']}). MeFormer is versatile to both conventional multi-label weather classification task and the proposed multi-weather co-presence estimation task, depending on the type of available ground truth.