Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Generalization Properties of Learning the Random Feature Models with Learnable Activation Functions

Zailin Ma, Jiansheng Yang, Yaodong Yang

TL;DR

The paper addresses the generalization limits of Random Feature models with Learnable Activation Functions (RFLAF) by deriving sharp data-dependent bounds on the required number of features $s$ under both plain and leverage-weighted sampling. It presents a unified theorem for arbitrary spectral measures, improves bounds for squared-error and Lipschitz losses, and proposes a practical algorithm to learn a weighted RFLAF via approximate kernel construction and leverage sampling. The main contributions include substantial reductions in the necessary feature count (e.g., from $\Omega(1/\epsilon^2)$ to $\widetilde{\Omega}(1/\epsilon)$ in certain settings) and improved sample-size and grid-size requirements, along with empirical validation showing similar performance with far fewer features. The results highlight the efficiency and scalability gains achievable by data-dependent feature sampling in RFLAF, with implications for fast kernel approximations and neural-style feature learning.

Abstract

This paper studies the generalization properties of a recently proposed kernel method, the Random Feature models with Learnable Activation Functions (RFLAF). By applying a data-dependent sampling scheme for generating features, we provide by far the sharpest bounds on the required number of features for learning RFLAF in both the regression and classification tasks. We provide a unified theorem that describes the complexity of the feature number $s$, and discuss the results for the plain sampling scheme and the data-dependent leverage weighted scheme. Through weighted sampling, the bound on $s$ in the MSE loss case is improved from $Ω(1/ε^2)$ to $\tildeΩ((1/ε)^{1/t})$ in general $(t\geq 1)$, and even to $Ω(1)$ when the Gram matrix has a finite rank. For the Lipschitz loss case, the bound is improved from $Ω(1/ε^2)$ to $\tildeΩ((1/ε^2)^{1/t})$. To learn the weighted RFLAF, we also propose an algorithm to find an approximate kernel and then apply the leverage weighted sampling. Empirical results show that the weighted RFLAF achieves the same performances with a significantly fewer number of features compared to the plainly sampled RFLAF, validating our theories and the effectiveness of this method.

On the Generalization Properties of Learning the Random Feature Models with Learnable Activation Functions

TL;DR

The paper addresses the generalization limits of Random Feature models with Learnable Activation Functions (RFLAF) by deriving sharp data-dependent bounds on the required number of features under both plain and leverage-weighted sampling. It presents a unified theorem for arbitrary spectral measures, improves bounds for squared-error and Lipschitz losses, and proposes a practical algorithm to learn a weighted RFLAF via approximate kernel construction and leverage sampling. The main contributions include substantial reductions in the necessary feature count (e.g., from to in certain settings) and improved sample-size and grid-size requirements, along with empirical validation showing similar performance with far fewer features. The results highlight the efficiency and scalability gains achievable by data-dependent feature sampling in RFLAF, with implications for fast kernel approximations and neural-style feature learning.

Abstract

This paper studies the generalization properties of a recently proposed kernel method, the Random Feature models with Learnable Activation Functions (RFLAF). By applying a data-dependent sampling scheme for generating features, we provide by far the sharpest bounds on the required number of features for learning RFLAF in both the regression and classification tasks. We provide a unified theorem that describes the complexity of the feature number , and discuss the results for the plain sampling scheme and the data-dependent leverage weighted scheme. Through weighted sampling, the bound on in the MSE loss case is improved from to in general , and even to when the Gram matrix has a finite rank. For the Lipschitz loss case, the bound is improved from to . To learn the weighted RFLAF, we also propose an algorithm to find an approximate kernel and then apply the leverage weighted sampling. Empirical results show that the weighted RFLAF achieves the same performances with a significantly fewer number of features compared to the plainly sampled RFLAF, validating our theories and the effectiveness of this method.
Paper Structure (35 sections, 15 theorems, 108 equations, 4 figures, 3 tables)

This paper contains 35 sections, 15 theorems, 108 equations, 4 figures, 3 tables.

Table of Contents

  1. INTRODUCTION
  2. PRELIMINARIES
  3. Target Function Class
  4. Random Feature Models with Learnable Activation Functions
  5. Ridge Leverage Score Function
  6. Weighted Sampling for Generating Random Features
  7. THEORETICAL RESULTS
  8. Learning with the Squared Error Loss
  9. Learning with a Lipschitz Continuous Loss
  10. Discussions of the Main Results
  11. Comparison with previous results
  12. Interpretation of the bounds
  13. Limitation of the results
  14. ALGORITHM
  15. Discussion of the time and space complexity
  16. ...and 20 more sections

Key Result

Lemma 2.1

For any $\epsilon>0$, if we set the width $h$ and the grid number $N$ as $\frac{1}{h}=\Omega\left(\frac{1}{\epsilon}\left(\log{\frac{1}{\epsilon}}\right)^{\frac{1}{2}}\right)$, $N=\Omega\left(\frac{1}{\epsilon^2}{\left(\log{\frac{1}{\epsilon}}\right)^\frac{3}{2}}\right)$, then there exists $\{\tilde

Figures (4)

  • Figure 1: Comparison of Leverage Weighted and Plain Sampling Schemes for RFLAF of Different Basis Functions (RBF in Line 1, B-spline in Line 2)
  • Figure 2: Comparison of Leverage Weighted and Plain Sampling Schemes for RFLAF of Different Grid Numbers ($N=32$ in Line 1, $N=64$ in Line 2)
  • Figure 3: Comparison of Leverage Weighted and Plain Sampling Schemes in Accuracies for RFLAF of Different Basis Functions (RBF in Line 1, B-spline in Line 2)
  • Figure 4: Comparison of Leverage Weighted and Plain Sampling Schemes in Accuracies for RFLAF of Different Grid Numbers ($N=16,32,64$ from top to bottom)

Theorems & Definitions (15)

  • Lemma 2.1: Proposition 4.1 in ma2024random
  • Theorem 3.2
  • Corollary 3.3: Leverage Weighted Sampling
  • Corollary 3.4: Plain Sampling
  • Theorem 3.6
  • Corollary 3.7: Leverage Weighted Sampling
  • Corollary 3.8: Plain Sampling
  • Theorem 4.1: Squared Error Loss
  • Theorem 4.2: Lipschitz Continuous Loss
  • Lemma E.1
  • ...and 5 more