Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

First-Mover Bias in Gradient Boosting Explanations: Mechanism, Detection, and Resolution

Drake Caraker, Bryan Arnold, David Rhoads

Abstract

We isolate and empirically characterize first-mover bias -- a path-dependent concentration of feature importance caused by sequential residual fitting in gradient boosting -- as a specific mechanistic cause of the well-known instability of SHAP-based feature rankings under multicollinearity. When correlated features compete for early splits, gradient boosting creates a self-reinforcing advantage for whichever feature is selected first: subsequent trees inherit modified residuals that favor the incumbent, concentrating SHAP importance on an arbitrary feature rather than distributing it across the correlated group. Scaling up a single model amplifies this effect -- a Large Single Model with the same total tree count as our method produces the worst explanations of any approach tested. We demonstrate that model independence is sufficient to resolve first-mover bias in the linear regime, and remains the most effective mitigation under nonlinear data-generating processes. Both our proposed method, DASH (Diversified Aggregation of SHAP), and simple seed-averaging (Stochastic Retrain) restore stability by breaking the sequential dependency chain, confirming that the operative mechanism is independence between explained models. At rho=0.9, both achieve stability=0.977, while the single-best workflow degrades to 0.958 and the Large Single Model to 0.938. On the Breast Cancer dataset, DASH improves stability from 0.32 to 0.93 (+0.61) against a tree-count-matched baseline. DASH additionally provides two diagnostic tools -- the Feature Stability Index (FSI) and Importance-Stability (IS) Plot -- that detect first-mover bias without ground truth, enabling practitioners to audit explanation reliability before acting on feature rankings. Software and reproducible benchmarks are available at https://github.com/DrakeCaraker/dash-shap.

First-Mover Bias in Gradient Boosting Explanations: Mechanism, Detection, and Resolution

Abstract

We isolate and empirically characterize first-mover bias -- a path-dependent concentration of feature importance caused by sequential residual fitting in gradient boosting -- as a specific mechanistic cause of the well-known instability of SHAP-based feature rankings under multicollinearity. When correlated features compete for early splits, gradient boosting creates a self-reinforcing advantage for whichever feature is selected first: subsequent trees inherit modified residuals that favor the incumbent, concentrating SHAP importance on an arbitrary feature rather than distributing it across the correlated group. Scaling up a single model amplifies this effect -- a Large Single Model with the same total tree count as our method produces the worst explanations of any approach tested. We demonstrate that model independence is sufficient to resolve first-mover bias in the linear regime, and remains the most effective mitigation under nonlinear data-generating processes. Both our proposed method, DASH (Diversified Aggregation of SHAP), and simple seed-averaging (Stochastic Retrain) restore stability by breaking the sequential dependency chain, confirming that the operative mechanism is independence between explained models. At rho=0.9, both achieve stability=0.977, while the single-best workflow degrades to 0.958 and the Large Single Model to 0.938. On the Breast Cancer dataset, DASH improves stability from 0.32 to 0.93 (+0.61) against a tree-count-matched baseline. DASH additionally provides two diagnostic tools -- the Feature Stability Index (FSI) and Importance-Stability (IS) Plot -- that detect first-mover bias without ground truth, enabling practitioners to audit explanation reliability before acting on feature rankings. Software and reproducible benchmarks are available at https://github.com/DrakeCaraker/dash-shap.
Paper Structure (78 sections, 9 equations, 5 figures, 9 tables, 1 algorithm)

This paper contains 78 sections, 9 equations, 5 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Why bigger models make it worse.
  3. Our contribution.
  4. Related Work
  5. SHAP and model explanations.
  6. Instability of feature importance.
  7. Ensemble explanations.
  8. Stable explanations.
  9. Explanation reliability and aggregation.
  10. Feature selection under multicollinearity.
  11. Problem Formulation
  12. Setup
  13. The instability problem
  14. Sequential residual dependency
  15. Empirical hypothesis.
  16. ...and 63 more sections

Figures (5)

  • Figure 1: The DASH five-stage pipeline. Models are trained independently (Stage 1), filtered for quality (Stage 2), selected for diversity (Stage 3), explained via TreeSHAP and averaged (Stage 4), and audited for stability (Stage 5).
  • Figure 2: First-mover bias visualized: per-feature importance within a correlated group ($\rho = 0.9$, true importance $= 0.40$ each). Single Best and LSM concentrate on an arbitrary feature; DASH distributes proportionally. Averaged over 5 repetitions; error bars show $\pm 1$ SD. The $\rho_S$ values in panel titles denote Spearman correlation with ground truth for this group (not the within-group feature correlation $\rho$).
  • Figure 3: Stability, accuracy, and equity as a function of within-group correlation $\rho$ (linear DGP, 20 repetitions per level). Independent methods (DASH, Stochastic Retrain) are flat across correlation levels; dependent methods (Single Best, LSM) degrade monotonically. All seven methods shown; Table \ref{['tab:sweep']} reports the four principal methods in detail, Table \ref{['tab:extended']} compares all at $\rho = 0.9$.
  • Figure 4: Diagnostic outputs on Breast Cancer. (a) The IS Plot identifies robust drivers (Quadrant I, green) and collinear cluster members (Quadrant II, red) without access to the correlation matrix. (b) The local disagreement map shows consensus SHAP values $\pm 1$ SD across the ensemble; wide error bars indicate model-dependent attributions.
  • Figure 5: Ablation sensitivity: stability as a function of population size $M$ at two correlation levels ($\rho = 0.0$ and $\rho = 0.9$). Stability is effectively invariant to population size (within 0.001 across $M \in \{50, 100, 200, 500\}$), indicating that $M = 200$ is a conservative default. Additional ablation results for $K$, $\varepsilon$, and $\delta$ are reported in Table \ref{['tab:epsilon']} and the inline text; full multi-panel ablation figures are planned for the journal version.