Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MMD-based Variable Importance for Distributional Random Forest

Clément Bénard, Jeffrey Näf, Julie Josse

TL;DR

Distributional Random Forest (DRF) estimates the full conditional distribution of a multivariate output, but standard variable importance focuses on mean effects. The paper introduces an $MMD$-based variable importance via a drop-and-relearn approach, defining $\mathrm{I}^{(j)}$ to measure changes in the distribution of $\mathbf{Y}$ when $X^{(-j)}$ is used, and provides a practical estimator $\mathrm{I}_n^{(j)}$ using DRF embeddings. A projected DRF variant is developed to reduce computational cost while preserving consistency. Theoretical guarantees show consistency of the estimators under standard kernel and data assumptions, and empirical results demonstrate superior performance over existing DRF measures, particularly in recursive feature elimination and high-dimensional settings.

Abstract

Distributional Random Forest (DRF) is a flexible forest-based method to estimate the full conditional distribution of a multivariate output of interest given input variables. In this article, we introduce a variable importance algorithm for DRFs, based on the well-established drop and relearn principle and MMD distance. While traditional importance measures only detect variables with an influence on the output mean, our algorithm detects variables impacting the output distribution more generally. We show that the introduced importance measure is consistent, exhibits high empirical performance on both real and simulated data, and outperforms competitors. In particular, our algorithm is highly efficient to select variables through recursive feature elimination, and can therefore provide small sets of variables to build accurate estimates of conditional output distributions.

MMD-based Variable Importance for Distributional Random Forest

TL;DR

Distributional Random Forest (DRF) estimates the full conditional distribution of a multivariate output, but standard variable importance focuses on mean effects. The paper introduces an -based variable importance via a drop-and-relearn approach, defining to measure changes in the distribution of when is used, and provides a practical estimator using DRF embeddings. A projected DRF variant is developed to reduce computational cost while preserving consistency. Theoretical guarantees show consistency of the estimators under standard kernel and data assumptions, and empirical results demonstrate superior performance over existing DRF measures, particularly in recursive feature elimination and high-dimensional settings.

Abstract

Distributional Random Forest (DRF) is a flexible forest-based method to estimate the full conditional distribution of a multivariate output of interest given input variables. In this article, we introduce a variable importance algorithm for DRFs, based on the well-established drop and relearn principle and MMD distance. While traditional importance measures only detect variables with an influence on the output mean, our algorithm detects variables impacting the output distribution more generally. We show that the introduced importance measure is consistent, exhibits high empirical performance on both real and simulated data, and outperforms competitors. In particular, our algorithm is highly efficient to select variables through recursive feature elimination, and can therefore provide small sets of variables to build accurate estimates of conditional output distributions.
Paper Structure (22 sections, 18 theorems, 78 equations, 6 figures, 8 tables)

This paper contains 22 sections, 18 theorems, 78 equations, 6 figures, 8 tables.

Table of Contents

  1. INTRODUCTION
  2. Context and Objectives.
  3. Motivating example.
  4. Contributions.
  5. DRF VARIABLE IMPORTANCE
  6. Theoretical importance measure.
  7. Variable importance estimate.
  8. THEORETICAL PROPERTIES
  9. Consistency of DRF importance.
  10. Projected DRF.
  11. EXPERIMENTS
  12. Simulated data
  13. Univariate output.
  14. Bivariate output.
  15. High-dimensional case.
  16. ...and 7 more sections

Key Result

Proposition 1

If $\mathrm{I}^{(j)}$ is the generalized total Sobol index defined by Equation (DRFvarimportance), then we have

Figures (6)

  • Figure 1: Values of $\mathrm{I}_n^{(1)}$, $\mathrm{I}_n^{(2)}$, and $\mathrm{I}_n^{(3)}$, with an increasing sample size for the bivariate output experiment.
  • Figure 2: RFE for 'rf1' (left panel) and 'wage' (right panel) datasets, using $\mathrm{I}_n^{(j)}$ (blue) or vimp-drf (red). The RFE procedure is repeated $40$ times to compute the standard error of the loss at each step, displayed as error bars.
  • Figure 3: RFE for 'enb' (left panel) and 'jura' (right panel) datasets, using our importance measure $\mathrm{I}_n^{(j)}$ (blue) or vimp-drf (red).
  • Figure 4: RFE for 'wq' (left panel) and 'scm20d' (right panel) datasets, using our importance measure $\mathrm{I}_n^{(j)}$ (blue) or vimp-drf (red).
  • Figure 5: RFE for 'Births' (left panel) and 'oes97' (right panel) datasets, using our importance measure $\mathrm{I}_n^{(j)}$ (blue) or vimp-drf (red).
  • ...and 1 more figures

Theorems & Definitions (29)

  • Proposition 1
  • Proposition 2
  • Proposition 3
  • Proposition 4
  • Theorem 1
  • Proposition 5
  • Theorem 2
  • Proposition 5
  • proof
  • Proposition 5
  • ...and 19 more