Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stabilizing black-box model selection with the inflated argmax

Melissa Adrian, Jake A. Soloff, Rebecca Willett

TL;DR

The paper tackles instability in black-box model selection by introducing a generic stabilization framework that combines bagging with an inflated argmax to produce a set of plausible models. This approach yields distribution-agnostic stability guarantees and adapts the number of returned models to the underlying uncertainty, without assuming strong data-model generative assumptions. The authors formalize model selection stability, present an explicit theoretical bound, and demonstrate the method across Lotka-Volterra dynamics, flow cytometry graphs, and kappa-means clustering, where it achieves stable, compact, and accurate model sets that outperform conventional baselines. This framework enables more trustworthy and interpretable model selection, with practical implications for scientific discovery and follow-up experimentation.

Abstract

Model selection is the process of choosing from a class of candidate models given data. For instance, methods such as the LASSO and sparse identification of nonlinear dynamics (SINDy) formulate model selection as finding a sparse solution to a linear system of equations determined by training data. However, absent strong assumptions, such methods are highly unstable: if a single data point is removed from the training set, a different model may be selected. In this paper, we present a new approach to stabilizing model selection with theoretical stability guarantees that leverages a combination of bagging and an ''inflated'' argmax operation. Our method selects a small collection of models that all fit the data, and it is stable in that, with high probability, the removal of any training point will result in a collection of selected models that overlaps with the original collection. We illustrate this method in (a) a simulation in which strongly correlated covariates make standard LASSO model selection highly unstable, (b) a Lotka-Volterra model selection problem focused on identifying how competition in an ecosystem influences species' abundances, (c) a graph subset selection problem using cell-signaling data from proteomics, and (d) unsupervised $κ$-means clustering. In these settings, the proposed method yields stable, compact, and accurate collections of selected models, outperforming a variety of benchmarks.

Stabilizing black-box model selection with the inflated argmax

TL;DR

The paper tackles instability in black-box model selection by introducing a generic stabilization framework that combines bagging with an inflated argmax to produce a set of plausible models. This approach yields distribution-agnostic stability guarantees and adapts the number of returned models to the underlying uncertainty, without assuming strong data-model generative assumptions. The authors formalize model selection stability, present an explicit theoretical bound, and demonstrate the method across Lotka-Volterra dynamics, flow cytometry graphs, and kappa-means clustering, where it achieves stable, compact, and accurate model sets that outperform conventional baselines. This framework enables more trustworthy and interpretable model selection, with practical implications for scientific discovery and follow-up experimentation.

Abstract

Model selection is the process of choosing from a class of candidate models given data. For instance, methods such as the LASSO and sparse identification of nonlinear dynamics (SINDy) formulate model selection as finding a sparse solution to a linear system of equations determined by training data. However, absent strong assumptions, such methods are highly unstable: if a single data point is removed from the training set, a different model may be selected. In this paper, we present a new approach to stabilizing model selection with theoretical stability guarantees that leverages a combination of bagging and an ''inflated'' argmax operation. Our method selects a small collection of models that all fit the data, and it is stable in that, with high probability, the removal of any training point will result in a collection of selected models that overlaps with the original collection. We illustrate this method in (a) a simulation in which strongly correlated covariates make standard LASSO model selection highly unstable, (b) a Lotka-Volterra model selection problem focused on identifying how competition in an ecosystem influences species' abundances, (c) a graph subset selection problem using cell-signaling data from proteomics, and (d) unsupervised -means clustering. In these settings, the proposed method yields stable, compact, and accurate collections of selected models, outperforming a variety of benchmarks.

Paper Structure

This paper contains 45 sections, 1 theorem, 30 equations, 15 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Our contributions
  3. Model selection
  4. Stage 1: Assigning weights to candidate models with A
  5. Stage 2: Conventional selection rules S
  6. Model selection stability
  7. Related work
  8. Our approach
  9. The inflated argmax for model selection
  10. Stability guarantee
  11. Experiments
  12. Model discovery of Lotka-Volterra dynamics
  13. Base algorithm A: Sparse Identification of Nonlinear Dynamical Systems (SINDy) with Sequentially Thresholded Ridge Regression (STRidge)
  14. Data generation
  15. Results
  16. ...and 30 more sections

Key Result

Theorem 1

(Adapted from Soloff2024.) For any model selection procedure $\mathcal{S}\circ\mathcal{A}$, our method $\textnormal{argmax}^\varepsilon \circ\widetilde{A}_{K, B}$ satisfies model selection stability at instability level where $\rho = \frac{K}{n}$ for subbagging and $\rho = 1-(1-\frac{1}{n})^K$ for bagging.

Figures (15)

  • Figure 1: Lotka-Volterra results. (Left column) Empirical CDF for the stability measures $\delta_j$, computed in \ref{['eq:delta_j']}, across $j\in [N]$ for each $\mathcal{S}$. We chose the parameters $\tau=0.63$, $k=2,$ and $\varepsilon = 0.09$ since these values yield a utility-weighted accuracy of approximately 0.3 for each $\mathcal{S}$. (Center column) Utility-weighted accuracy, computed in \ref{['eq:utility_acc']}, versus the worst-case instability across $\delta_j$. (Right column) Median number of models returned across $j$ versus the maximum $\delta_j$ across $j\in[N]$ for each $\mathcal{S}$. (Center and right columns) We plot the range of values $k \in [1,\dots, 6]$, and $\epsilon\in(0,1)$ and $\tau\in(0,1)$ with approximately evenly spread values across their supports.
  • Figure 2: Empirical correlation matrix of the 11 proteins. Data from Sachs2005.
  • Figure 3: Visualization of the top two graph structures selected via subbagged gLASSO with 10,000 bags. The red connection shown in the right graph highlights that this connection is the only difference between the top 1 and top 2 selected graph structures. The top 1 graph was selected for 9.31% of bags, and the top 2 graph was selected for 8.54% of bags.
  • Figure 4: Unsupervised $\kappa$-means clustering results. (Left column) Empirical CDF for the stability measures $\delta_j$, computed in \ref{['eq:delta_j']}, across $j\in [N]$ for each $\mathcal{S}$. We chose the parameters $k=2,$ and $\varepsilon = 0.3$. (Center column) Utility-weighted accuracy, computed in \ref{['eq:utility_acc']}, versus the worst-case instability across $\delta_j$. (Right column) Median number of models returned across $j$ versus the maximum $\delta_j$ across $j\in[N]$ for each $\mathcal{S}$. (Center and right columns) We plot the range of values $k \in [1,2,3]$, and $\epsilon\in[0.005, 0.01 ,0.05, 0.1, 0.2, 0.3, 0.5, 0.7, 0.95,1.0]$.
  • Figure 5: Covariance matrix for each simulated dataset. Covariate indices 1 and 3 were used to construct the response $y$.
  • ...and 10 more figures

Theorems & Definitions (3)

  • Definition 2.1
  • Definition 4.1: Inflated argmax
  • Theorem 1