Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Curve Your Enthusiasm: Concurvity Regularization in Differentiable Generalized Additive Models

Julien Siems, Konstantin Ditschuneit, Winfried Ripken, Alma Lindborg, Maximilian Schambach, Johannes S. Otterbach, Martin Genzel

TL;DR

This work identifies concurvity as a key obstacle to the interpretability of differentiable GAMs and proposes a simple, differentiable regularizer $R_{\perp}$ that penalizes pairwise correlations among the non-linear feature mappings $f_i(X_i)$. By optimizing $\min_{\beta,(f_i)} \mathbb{E}[L(Y,\beta+\sum f_i(X_i))] + \lambda R_{\perp}(\{f_i\},\{X_i\})$, the approach decorrelates additive components, improving interpretability without severely compromising predictive accuracy. Empirical results across toy, time-series, and tabular datasets (including Neural Additive Models and NeuralProphet) show reduced concurvity, more stable feature importances, and clearer component separation, with a moderate trade-off controlled by $\lambda$. These findings suggest that decorrelation-focused regularization can enhance the reliability and transparency of GAM-based models in safety-critical and regulated settings.

Abstract

Generalized Additive Models (GAMs) have recently experienced a resurgence in popularity due to their interpretability, which arises from expressing the target value as a sum of non-linear transformations of the features. Despite the current enthusiasm for GAMs, their susceptibility to concurvity - i.e., (possibly non-linear) dependencies between the features - has hitherto been largely overlooked. Here, we demonstrate how concurvity can severly impair the interpretability of GAMs and propose a remedy: a conceptually simple, yet effective regularizer which penalizes pairwise correlations of the non-linearly transformed feature variables. This procedure is applicable to any differentiable additive model, such as Neural Additive Models or NeuralProphet, and enhances interpretability by eliminating ambiguities due to self-canceling feature contributions. We validate the effectiveness of our regularizer in experiments on synthetic as well as real-world datasets for time-series and tabular data. Our experiments show that concurvity in GAMs can be reduced without significantly compromising prediction quality, improving interpretability and reducing variance in the feature importances.

Curve Your Enthusiasm: Concurvity Regularization in Differentiable Generalized Additive Models

TL;DR

This work identifies concurvity as a key obstacle to the interpretability of differentiable GAMs and proposes a simple, differentiable regularizer that penalizes pairwise correlations among the non-linear feature mappings . By optimizing , the approach decorrelates additive components, improving interpretability without severely compromising predictive accuracy. Empirical results across toy, time-series, and tabular datasets (including Neural Additive Models and NeuralProphet) show reduced concurvity, more stable feature importances, and clearer component separation, with a moderate trade-off controlled by . These findings suggest that decorrelation-focused regularization can enhance the reliability and transparency of GAM-based models in safety-critical and regulated settings.

Abstract

Generalized Additive Models (GAMs) have recently experienced a resurgence in popularity due to their interpretability, which arises from expressing the target value as a sum of non-linear transformations of the features. Despite the current enthusiasm for GAMs, their susceptibility to concurvity - i.e., (possibly non-linear) dependencies between the features - has hitherto been largely overlooked. Here, we demonstrate how concurvity can severly impair the interpretability of GAMs and propose a remedy: a conceptually simple, yet effective regularizer which penalizes pairwise correlations of the non-linearly transformed feature variables. This procedure is applicable to any differentiable additive model, such as Neural Additive Models or NeuralProphet, and enhances interpretability by eliminating ambiguities due to self-canceling feature contributions. We validate the effectiveness of our regularizer in experiments on synthetic as well as real-world datasets for time-series and tabular data. Our experiments show that concurvity in GAMs can be reduced without significantly compromising prediction quality, improving interpretability and reducing variance in the feature importances.
Paper Structure (41 sections, 1 theorem, 11 equations, 19 figures, 2 tables)

This paper contains 41 sections, 1 theorem, 11 equations, 19 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Generalized Additive Models
  4. Multicollinearity and Concurvity
  5. Concurvity Regularizer
  6. Experimental Evaluation
  7. Toy Examples
  8. Toy Example 1: Concurvity regularization with and without multicollinearity
  9. Toy Example 2: Concurvity regularization in the case of non-linearly dependent features
  10. Time-Series Data
  11. Tabular Data
  12. Case study: California Housing
  13. Related Work
  14. Classical works on concurvity in GAMs
  15. Modern neural approaches to GAMs
  16. ...and 26 more sections

Key Result

Lemma A.1

Let $X_1, \dots, X_p \in \mathbb{R}^{N}$ be a set of feature variables with $p > 1$ and let $\mathcal{H} \subset \{ (f_1, \dots, f_p) \mid f_i: \mathbb{R} \to \mathbb{R} \}$ be a class of functions. Consider the following subclass of $\mathcal{H}$: Then we do not have concurvity w.r.t. $X_1, \dots, X_p$ and $\mathcal{H}_{\perp}$.

Figures (19)

  • Figure 1: Concurvity in a NeuralProphet model: Fitting a time series composed of daily and weekly seasonalities, each represented by Fourier terms. (left) Using few Fourier terms results in uncorrelated components but a poor fit. (middle) A more complex model improves the fit but sacrifices interpretability due to self-canceling high-frequency terms. (right) The same complex model, but with our regularizer, achieves both good predictive performance and interpretable (decorrelated) components. See Section \ref{['subsec:experiments:prophet']} for more details.
  • Figure 2: Results for Toy Example 1. (a) Effect of concurvity regularization on uncorrelated and correlated features. For each of the six settings, 40 random initializations were evaluated. (b) Trade-off curve between model accuracy (validation RMSE) and measured concurvity (validation $R_{\perp}$). Results are averaged over 10 random initializations per regularization strength $\lambda$.
  • Figure 3: Results for Toy Example 2. (a) Comparison of transformed feature correlation with and without concurvity regularization. (b) Trade-off curve between model accuracy (validation RMSE) and measured concurvity ($R_{\perp}$).
  • Figure 4: Trade-off curve for NeuralProphet model trained on step-function data.
  • Figure 5: Trade-off curves between model fit quality and measured concurvity $R_{\perp}$ for 50 levels of concurvity regularization strength $\lambda$. Each regularization strength is evaluated over 10 initialization seeds to account for training variability, particularly noticeable in smaller datasets. The results of a conventional GAM are shown for comparison.
  • ...and 14 more figures

Theorems & Definitions (4)

  • Definition 2.1: Multicollinearity
  • Definition 2.2: Concurvity
  • Lemma A.1
  • proof