Example-Based Concept Analysis Framework for Deep Weather Forecast Models

TL;DR

The paper addresses the challenge of trusting deep weather forecast models by introducing an example-based concept analysis framework that surfaces conceptually analogous cases from model inferences. It combines a Probabilistic Concept Prober with a Nearest-Neighbor Search Engine, grounded by a human-annotated concept dataset and a domain-specific UI, to translate internal representations into meteorologically meaningful explanations. Experiments on a DeepRaNE-based rainfall segmentation model demonstrate that the concepts align with expert perceptions, and the framework supports model debugging through uncertainty estimation and actionable visualizations. The work advances practical, domain-aware XAI for weather forecasting, enabling better interpretability, debugging, and potential operational adoption.

Abstract

To improve the trustworthiness of an AI model, finding consistent, understandable representations of its inference process is essential. This understanding is particularly important in high-stakes operations such as weather forecasting, where the identification of underlying meteorological mechanisms is as critical as the accuracy of the predictions. Despite the growing literature that addresses this issue through explainable AI, the applicability of their solutions is often limited due to their AI-centric development. To fill this gap, we follow a user-centric process to develop an example-based concept analysis framework, which identifies cases that follow a similar inference process as the target instance in a target model and presents them in a user-comprehensible format. Our framework provides the users with visually and conceptually analogous examples, including the probability of concept assignment to resolve ambiguities in weather mechanisms. To bridge the gap between vector representations identified from models and human-understandable explanations, we compile a human-annotated concept dataset and implement a user interface to assist domain experts involved in the the framework development.

Figures (10)

• Figure 1: Illustration of concept prober training process.
• Figure 2: Examples of neighbor search engine and probabilistic concept explanations. Each row is a single query. The first column of each row are the query samples. The remaining columns are the three nearest neighbors of the queries. Each instance is reported with top-5 rainfall mechanism concepts in terms of prober's probability. The numerical values after each concept are the predictive probabilities from the corresponding prober.
• Figure 3: The importance score of concept activation vectors for each concept. The left panel displays the scores with respect to each target class, while the right panel represents the scores with respect to the loss value, i.e., encompassing all classes. The numbers preceding the concept labels indicate the label indices.
• Figure 4: Perturbation test of concept activation vectors. $\hat{y}, y,$ and $\tilde{y}'$ denote one hour ahead prediction, the ground truth, and the perturbed predictions, respectively. "Examples of concept 6 (easterlies rainfall) and concept 10 (isolated) illustrate the nonlinear development effect on future predictions while retaining their underlying mechanisms."
• Figure 5: Probabilities and uncertainties of concept probers on bright band effect and light rainfall cases. Each concept explanation is accompanied by its predictive probability (shown as the left number) and its uncertainty (in parentheses).