Approximation and generalization properties of the random projection classification method

TL;DR

The paper analyzes a remarkably simple classifier family that thresholds random one-dimensional projections of polynomial-extended features. It proves generalization-gap bounds that are independent of the ambient dimension and the polynomial degree, and shows these bounds can outperform VC-based guarantees for moderate numbers of projections. It further establishes universal approximation properties: with sufficiently large $n$ and $k$, the sign of random polynomials can approximate any continuous function on a compact set, implying near-Bayes performance when class-conditionals are known. Additionally, chaining yields tighter asymptotic bounds, and for certain problems with a large Rashomon ratio, very few projections suffice in practice, underscoring the method’s potential for robust, scalable high-dimensional classification.

Abstract

The generalization gap of a classifier is related to the complexity of the set of functions among which the classifier is chosen. We study a family of low-complexity classifiers consisting of thresholding a random one-dimensional feature. The feature is obtained by projecting the data on a random line after embedding it into a higher-dimensional space parametrized by monomials of order up to k. More specifically, the extended data is projected n-times and the best classifier among those n, based on its performance on training data, is chosen. We show that this type of classifier is extremely flexible as, given full knowledge of the class conditional densities, under mild conditions, the error of these classifiers would converge to the optimal (Bayes) error as k and n go to infinity. We also bound the generalization gap of the random classifiers. In general, these bounds are better than those for any classifier with VC dimension greater than O(ln n). In particular, the bounds imply that, unless the number of projections n is extremely large, the generalization gap of the random projection approach is significantly smaller than that of a linear classifier in the extended space. Thus, for certain classification problems (e.g., those with a large Rashomon ratio), there is a potntially large gain in generalization properties by selecting parameters at random, rather than selecting the best one amongst the class.

Figures (2)

• Figure 1: A comparison between the bounds for the expectation of the generalization gap of the method of thresholding after $n$ random projections and an algorithm with VC dimension $d_{VC}=2$ and $d_{VC}=3$, respectively. The graph is generated under the assumption that $N=10000$.
• Figure 2: Empirical generalization gap for different mixture levels of data. Parameters are: $d=65$, $\boldsymbol{p}^{1}=(0.25,...,0.8,1)$, at $x=20$$\boldsymbol{p}^{-1}=(0.8,...,0.25,1)$, $N_{\text{train}}=200$, $N_{\text{test}}=2000$, green line with pentagons correspond to thresholding after random projection with $n=10000$, blue line with triangles corresponds to logistic regression, red line with crosses corresponds to linear SVM and orange line with circles corresponds to SVM with Gaussian kernel. The left and right figures differ in the level of noise: (a) $\sigma=0$ (b) $\sigma=0.05$. The theoretical bounds for the generalization gap are 2.7 for the linear models (logistic regression and SVM) and 0.8 for thresholding after random projection). Note that the distribution of the data represented on the horizontal axis differs only in the labels - the distribution of independent variables $\boldsymbol{x}$ stays the same. Therefore this experiment shows amongst other things that the generalization bounds should take labels into account even for simple models (similar issue for the case of deep models is discussed in recht_rethinking).

Theorems & Definitions (20)

• Theorem 1
• proof
• Corollary 1
• proof
• Theorem 2
• proof
• Corollary 2
• proof
• Theorem 3
• proof