Revisiting Randomization in Greedy Model Search

TL;DR

The paper studies how feature subsampling interacts with greedy forward selection under a fixed, orthogonal design. It shows that ensemble forward selection (EFS) can simultaneously reduce both bias and variance, with training error and degrees of freedom that are not monotone in the subsampling rate and coefficients reweighted in a logistic-like manner by rank. The authors derive exact weight recurrences for finite samples and establish a logistic approximation in the asymptotic regime where m/p → γ, revealing that algorithmic regularization from randomization acts differently when combined with greedy optimization than with convex learners. These results extend the theory of randomized ensembles beyond ridge-like shrinkage, connect to elastic-net and lasso behavior, and highlight practical implications for improving greedy model search through controlled feature subsampling.

Abstract

Feature subsampling is a core component of random forests and other ensemble methods. While recent theory suggests that this randomization acts solely as a variance reduction mechanism analogous to ridge regularization, these results largely rely on base learners optimized via ordinary least squares. We investigate the effects of feature subsampling on greedy forward selection, a model that better captures the adaptive nature of decision trees. Assuming an orthogonal design, we prove that ensembling with feature subsampling can reduce both bias and variance, contrasting with the pure variance reduction of convex base learners. More precisely, we show that both the training error and degrees of freedom can be non-monotonic in the subsampling rate, breaking the analogy with standard shrinkage methods like the lasso or ridge regression. Furthermore, we characterize the exact asymptotic behavior of the estimator, showing that it adaptively reweights OLS coefficients based on their rank, with weights that are well-approximated by a logistic function. These results elucidate the distinct role of algorithmic randomization when interleaved with greedy optimization.

Figures (2)

• Figure 1: Comparison of $\tilde{w}_{j}^{k,\gamma}$ (solid) and the logistic approximation \ref{['eq:logistic']} (dashed) for $\gamma=0.05, 0.15,$ and $0.333$, with $k=10$.
• Figure 2: Degrees of freedom vs. training MSE for EFS($k,m$) and FS($k$) with $p/n = 0.1$ and banded exact sparsity, as $k$ varies over $\{1,2,\ldots,20\}$ (true sparsity is 10). At each $k$, EFS($k,m$) reduces both quantities, shifting the tradeoff curve toward the origin. Gains are more pronounced at low SNR. At high SNR, EFS($k,m$) and FS($k$) become similar.

Theorems & Definitions (38)

• Definition 3.1: Feature ordering
• Lemma 3.1: Monotonicity
• Lemma 3.2: Recurrence relation
• proof : Proof sketch of Lemma \ref{['lem:weights_monotonicity']}
• proof : Proof sketch of Lemma \ref{['prop:finitep']}
• Theorem 3.1: Asymptotic weights for fixed $k$
• proof : Proof sketch
• Theorem 3.2: Asymptotic weights for large $k$
• proof : Proof sketch
• Theorem 4.1: Training error improvement