Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How Understanding Forecast Uncertainty Resolves the Explainability Problem in Machine Learning Models

Joseph L. Breeden

TL;DR

This work reframes explainability as a function of forecast usability, showing that high explainability near decision boundaries is illusory if forecast uncertainty is large. It introduces two uncertainty measures—Local Linear Uncertainty and Conformal Uncertainty—together with Lipschitz, Jaccard, and Hessian-based instability metrics to quantify when local explanations are reliable. Theoretical derivations connect gradient magnitude and curvature to both forecast uncertainty and explanation instability, supported by simulations on nonlinear and piecewise-linear functions. Practically, the authors advocate assessing forecast uncertainty first and applying safe fallbacks (e.g., logistic regression) in high-uncertainty regions, arguing for uncertainty-aware decision workflows in credit underwriting and other high-stakes domains.

Abstract

For applications of machine learning in critical decisions, explainability is a primary concern, and often a regulatory requirement. Local linear methods for generating explanations, such as LIME and SHAP, have been criticized for being unstable near decision boundaries. In this paper, we explain that such concerns reflect a misunderstanding of the problem. The forecast uncertainty is high at decision boundaries, so consequently, the explanatory instability is high. The correct approach is to change the sequence of events and questions being asked. Nonlinear models can be highly predictive in some regions while having little or no predictability in others. Therefore, the first question is whether a usable forecast exists. When there is a forecast with low enough uncertainty to be useful, an explanation can be sought via a local linear approximation. In such cases, the explanatory instability is correspondingly low. When no usable forecast exists, the decision must fall to a simpler overall model such as traditional logistic regression. Additionally, these results show that some methods that purport to be explainable everywhere, such as ReLU networks or any piecewise linear model, have only an illusory explainability, because the forecast uncertainty at the segment boundaries is too high to be useful. Explaining an unusable forecast is pointless.

How Understanding Forecast Uncertainty Resolves the Explainability Problem in Machine Learning Models

TL;DR

This work reframes explainability as a function of forecast usability, showing that high explainability near decision boundaries is illusory if forecast uncertainty is large. It introduces two uncertainty measures—Local Linear Uncertainty and Conformal Uncertainty—together with Lipschitz, Jaccard, and Hessian-based instability metrics to quantify when local explanations are reliable. Theoretical derivations connect gradient magnitude and curvature to both forecast uncertainty and explanation instability, supported by simulations on nonlinear and piecewise-linear functions. Practically, the authors advocate assessing forecast uncertainty first and applying safe fallbacks (e.g., logistic regression) in high-uncertainty regions, arguing for uncertainty-aware decision workflows in credit underwriting and other high-stakes domains.

Abstract

For applications of machine learning in critical decisions, explainability is a primary concern, and often a regulatory requirement. Local linear methods for generating explanations, such as LIME and SHAP, have been criticized for being unstable near decision boundaries. In this paper, we explain that such concerns reflect a misunderstanding of the problem. The forecast uncertainty is high at decision boundaries, so consequently, the explanatory instability is high. The correct approach is to change the sequence of events and questions being asked. Nonlinear models can be highly predictive in some regions while having little or no predictability in others. Therefore, the first question is whether a usable forecast exists. When there is a forecast with low enough uncertainty to be useful, an explanation can be sought via a local linear approximation. In such cases, the explanatory instability is correspondingly low. When no usable forecast exists, the decision must fall to a simpler overall model such as traditional logistic regression. Additionally, these results show that some methods that purport to be explainable everywhere, such as ReLU networks or any piecewise linear model, have only an illusory explainability, because the forecast uncertainty at the segment boundaries is too high to be useful. Explaining an unusable forecast is pointless.
Paper Structure (30 sections, 29 equations, 2 figures)

This paper contains 30 sections, 29 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Counterfactual Explanations and Algorithmic Recourse
  4. Limits of Post-hoc Explanations in High-stakes Decisions
  5. Calibration and Decision-grade Probabilities
  6. Model Risk Management Context
  7. Forecast Uncertainty Estimation Methods
  8. Measures of Explanatory Stability
  9. Quantifying Forecast Uncertainty
  10. Local Linear Uncertainty
  11. Conformal Uncertainty Estimation
  12. Quantifying Explanatory Instability
  13. Lipschitz-Based Local Sensitivity Metric
  14. Jaccard Top-k Feature Overlap Metric
  15. Hessian Instability
  16. ...and 15 more sections

Figures (2)

  • Figure 1: The relationship between Hessian magnitude explanatory instability and local linear uncertainty. Each point is the estimate at one randomly sampled point in the feature space.
  • Figure 2: The relationship between Lipschitz explanatory instability and conformal forecast uncertainty. Each point is the estimate at one randomly sampled point in the feature space.