Learning with Subset Stacking

TL;DR

This work introduces LESS, a regression algorithm that learns from localized subsets and aggregates their predictions through a global model to address heterogeneity in input-output relationships. By constructing a distance-weighted feature vector from local predictors and training a two-layer global learner, LESS can be executed in parallel and adapted via averaging or boosting (LESS-A/LESS-B). Empirical results on several UCI datasets show LESS is competitive with, and often superior to, established baselines, with analyses highlighting the importance of weighting and the global-learning step. The approach offers a scalable, interpretable framework with potential extensions to deterministic subset design, scalable global learners, and classification.

Abstract

We propose a new regression algorithm that learns from a set of input-output pairs. Our algorithm is designed for populations where the relation between the input variables and the output variable exhibits a heterogeneous behavior across the predictor space. The algorithm starts with generating subsets that are concentrated around random points in the input space. This is followed by training a local predictor for each subset. Those predictors are then combined in a novel way to yield an overall predictor. We call this algorithm "LEarning with Subset Stacking" or LESS, due to its resemblance to the method of stacking regressors. We offer bagging and boosting variants of LESS and test against the state-of-the-art methods on several datasets. Our comparison shows that LESS is highly competitive.

Figures (8)

• Figure 1: Schematic view of the basic LESS algorithm.
• Figure 2: Demonstration of local models in LESS on one-dimensional synthetic dataset.
• Figure 3: The effects of the number of replications ($b$), the number of subsets ($m$), and the number of neighbors ($k$). Notice that $n = m \times k$. The subsets are constructed by randomly selecting $m$ anchor points.
• Figure 4: Results of the ablation study. The vertical axis is obtained by scaling the MSE values with the maximum one among NoW-NoG (no weighting, no global learning), W-NoG (weighting, no global learning), NoW-G (no weighting, global learning), LESS (default).
• Figure 5: Computation times of LESS-A and LESS-B (middle) vs number of subsets. Right: Time comparison against other methods; the horizontal dashed line shows the computation times of LESS-A and LESS-B with its default value of 5% of the samples used for setting the number of subsets to 20.