Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Challenges in interpretability of additive models

Xinyu Zhang, Julien Martinelli, ST John

TL;DR

This paper surveys generalized additive models and neural additive models as transparent, interpretable alternatives in machine learning, while arguing that interpretability is not guaranteed. It reviews the formal GAM/NAM framework, learning strategies for shape functions, and common interpretability metrics, and it discusses extensions such as higher-order interactions and uncertainty estimation. A central focus is nonidentifiability: sum-only observations and concurvity lead to indeterminacy and multiple plausible explanations (Rashomon effects), challenging straightforward interpretation. The authors advocate caution in claims of interpretability or safety-critical suitability, and propose embracing ensemble Rashomon-set approaches and domain-guided criteria to convey multiple plausible explanations rather than a single definitive one.

Abstract

We review generalized additive models as a type of ``transparent'' model that has recently seen renewed interest in the deep learning community as neural additive models. We highlight multiple types of nonidentifiability in this model class and discuss challenges in interpretability, arguing for restraint when claiming ``interpretability'' or ``suitability for safety-critical applications'' of such models.

Challenges in interpretability of additive models

TL;DR

This paper surveys generalized additive models and neural additive models as transparent, interpretable alternatives in machine learning, while arguing that interpretability is not guaranteed. It reviews the formal GAM/NAM framework, learning strategies for shape functions, and common interpretability metrics, and it discusses extensions such as higher-order interactions and uncertainty estimation. A central focus is nonidentifiability: sum-only observations and concurvity lead to indeterminacy and multiple plausible explanations (Rashomon effects), challenging straightforward interpretation. The authors advocate caution in claims of interpretability or safety-critical suitability, and propose embracing ensemble Rashomon-set approaches and domain-guided criteria to convey multiple plausible explanations rather than a single definitive one.

Abstract

We review generalized additive models as a type of ``transparent'' model that has recently seen renewed interest in the deep learning community as neural additive models. We highlight multiple types of nonidentifiability in this model class and discuss challenges in interpretability, arguing for restraint when claiming ``interpretability'' or ``suitability for safety-critical applications'' of such models.

Paper Structure

This paper contains 15 sections, 9 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background on additive models
  3. General definition
  4. Literature review: modeling and learning shape functions
  5. How we interpret additive models
  6. Extensions of additive models
  7. Incorporating higher-order interactions
  8. Modeling "jagged" data
  9. Uncertainty estimation
  10. Nonidentifiability
  11. Nonidentifiability due to only observing sum of terms
  12. Nonidentifiability due to concurvity/correlated covariates
  13. Single identifiable model does not guarantee full insight into data under concurvity
  14. Uncorrelated shape functions do not guarantee understandable marginal effects
  15. Conclusions

Figures (2)

  • Figure 1: Shape functions learned by fitting NAMs on the subsets $\{X_1,X_5, X_6\}$ (top) and $\{X_1, X_3, X_6\}$ (bottom) to data generated from \ref{['eq:kovacs-example']}.
  • Figure 2: Shape functions learned by fitting NAMs with different concurvity regularization strength to data generated from \ref{['eqn:interaction-example']}.