Physically Interpretable Probabilistic Domain Characterization

TL;DR

A method to predict the likelihood of different weather conditions from images captured by vehicle-mounted cameras by estimating distributions of physical parameters using normalizing flows and evaluating whether a system can safely operate in a target domain by comparing it to multiple source domains where safety has already been established is developed.

Abstract

Characterizing domains is essential for models analyzing dynamic environments, as it allows them to adapt to evolving conditions or to hand the task over to backup systems when facing conditions outside their operational domain. Existing solutions typically characterize a domain by solving a regression or classification problem, which limits their applicability as they only provide a limited summarized description of the domain. In this paper, we present a novel approach to domain characterization by characterizing domains as probability distributions. Particularly, we develop a method to predict the likelihood of different weather conditions from images captured by vehicle-mounted cameras by estimating distributions of physical parameters using normalizing flows. To validate our proposed approach, we conduct experiments within the context of autonomous vehicles, focusing on predicting the distribution of weather parameters to characterize the operational domain. This domain is characterized by physical parameters (absolute characterization) and arbitrarily predefined domains (relative characterization). Finally, we evaluate whether a system can safely operate in a target domain by comparing it to multiple source domains where safety has already been established. This approach holds significant potential, as accurate weather prediction and effective domain adaptation are crucial for autonomous systems to adjust to dynamic environmental conditions.

Figures (10)

• Figure 1: The two synthetic images, A and C, generated using the CARLA software, appear almost identical, despite being acquired under very different weather conditions. This highlights the challenge of information loss from sensors like cameras when dealing with weather-related physical parameters. As a result, predictions that diverge from the ground truth in such ambiguous cases should not be penalized during evaluation.
• Figure 2: Considering the images A and C from Fig. \ref{['fig:observation-1-ambiguous-cases']}, generating an image B for the arithmetically averaged parameters, leads to an image very different from A and C. In other words, there are images $i$ such that the probability $P(I=i,W=\hat{w}(i))$ of the pair (image, estimated weather) is zero when the estimated weather is the expected value of the weather knowing the image, $\hat{w}(i)=E[W|I=i]$, as done in regression. Working with distributions of weather parameters (as shown using a contour plot on the right-hand side) as proposed in this work, rather than predicting specific values for each parameter, , regression (as shown on the left-hand side), avoids this problem.
• Figure 3: Task I. The aim of the first task is, based on an image, to predict the joint distribution of the weather parameters. For this purpose, (1) we generate data using the CARLA software for uniformly distributed weather parameters (offline), (2) we train a NPE model using the learning set (LS) of our generated data (offline), and (3) we infer, given an image from the test set (TS), the estimated weather distribution (normalizing flow) and show the result on a corner plot.
• Figure 4: Excerpt of the images in our dataset generated with the CARLA simulator. The ground-truth weather parameters are drawn at random for each image, following a uniform distribution with the bounds given in Table \ref{['tbl:weather-parameters']}.
• Figure 5: Task I: results obtained, with the model learned with the ResNet-50 backbone on $500$k learning samples, for an arbitrarily chosen input image in the TS.