Knowledge-Guided Machine Learning: Illustrating the use of Explainable Boosting Machines to Identify Overshooting Tops in Satellite Imagery

TL;DR

It is demonstrated that EBMs are particularly suitable to implement human-guided strategies in an ML algorithm, and a fully interpretable ML algorithm developed in a human-machine collaboration that uses human-guided strategies is demonstrated.

Abstract

Machine learning (ML) algorithms have emerged in many meteorological applications. However, these algorithms struggle to extrapolate beyond the data they were trained on, i.e., they may adopt faulty strategies that lead to catastrophic failures. These failures are difficult to predict due to the opaque nature of ML algorithms. In high-stakes applications, such as severe weather forecasting, is is crucial to avoid such failures. One approach to address this issue is to develop more interpretable ML algorithms. The primary goal of this work is to illustrate the use of a specific interpretable ML algorithm that has not yet found much use in meteorology, Explainable Boosting Machines (EBMs). We demonstrate that EBMs are particularly suitable to implement human-guided strategies in an ML algorithm. As guiding example, we show how to develop an EBM to detect overshooting tops (OTs) in satellite imagery. EBMs require input features to be scalar. We use techniques from Knowledge-Guided Machine Learning to first extract scalar features from meteorological imagery. For the application of identifying OTs this includes extracting cloud texture from satellite imagery using Gray-Level Co-occurrence Matrices. Once trained, the EBM was examined and minimally altered to more closely match strategies used by domain scientists to identify OTs. The result of our efforts is a fully interpretable ML algorithm developed in a human-machine collaboration that uses human-guided strategies. While the final model does not reach the accuracy of more complex approaches, it performs reasonably well and we hope paves the way for building more interpretable ML algorithms for this and other meteorological applications.

Figures (18)

• Figure 1: Illustration of Clever Hans Strategies by lapuschkin2019unmasking. These examples illustrate three strategies of an AI algorithm tasked to decide whether a given image contains a horse. Input images of the algorithm are shown on top. The corresponding attribution maps generated by Layer-Wise Relevance Propagation are shown on the bottom. These maps are overlays indicating which pixels in the original image contribute positively (red) or negatively (blue) to the AI system deciding that the image contains a horse. Strategy 1 is appropriate, while Strategies 2 and 3 are faulty. Image credit: Adapted from Fig. 3 in lapuschkin2019unmasking.
• Figure 2: Examples of two ML workflows--(a) for black-box ML models, such as neural networks, and (b) for EBMs. Note that in (b) the model's strategies are modified without re-training the model.
• Figure 3: Examples of (a) visible channel reflectance, (b) infrared channel brightness temperature, (c) MRMS labels (in red), and (d) IR/VIS sandwich product, each from 5 June 2024 at 21:45:00Z.
• Figure 4: Summarized approach displaying (a) the satellite imagery used to generate the three features used in our approach (taken 5 June 2024 at 21:45:00Z), (b) the three features themselves (first row: brightness, second row: infrared followed by cool contrast tiles), (c) the interpretable machine learning method used (EBM) and a visualization of three of its learned strategies, and (d) a map of OT locations (in red) as detected by the finalized model overlaid on the corresponding visible imagery.
• Figure 5: Brightness feature example. Derived from VIS imagery (for this figure, taken 5 June 2024 at 21:45:00Z). This scene represents the blurred and downsampled version of the VIS imagery displayed in Fig. \ref{['VisibleInfraredMRMS']}a.