Manifold regularization based on Nystr{ö}m type subsampling
Abhishake Rastogi, Sivananthan Sampath
TL;DR
This work tackles the computational challenge of large-scale kernel-based multi-task learning by employing Nyström-type subsampling within a vector-valued RKHS framework and a multi-penalty regularization scheme. A linear functional strategy aggregates multiple Nyström approximants to form a robust estimator, with theoretical guarantees attaining minimax rates via an effective dimension concept. The analysis combines operator-valued kernels, sampling operators, and probabilistic bounds to establish convergence in both RKHS and $\mathscr{L}^2_{\rho}$ norms. Empirically, the approach demonstrates strong performance and scalability on Caltech-101 and NSL-KDD datasets, showing that aggregation can match or exceed standard Nyström methods while significantly reducing computation.
Abstract
In this paper, we study the Nystr{ö}m type subsampling for large scale kernel methods to reduce the computational complexities of big data. We discuss the multi-penalty regularization scheme based on Nystr{ö}m type subsampling which is motivated from well-studied manifold regularization schemes. We develop a theoretical analysis of multi-penalty least-square regularization scheme under the general source condition in vector-valued function setting, therefore the results can also be applied to multi-task learning problems. We achieve the optimal minimax convergence rates of multi-penalty regularization using the concept of effective dimension for the appropriate subsampling size. We discuss an aggregation approach based on linear function strategy to combine various Nystr{ö}m approximants. Finally, we demonstrate the performance of multi-penalty regularization based on Nystr{ö}m type subsampling on Caltech-101 data set for multi-class image classification and NSL-KDD benchmark data set for intrusion detection problem.