Sparse Representer Theorems for Learning in Reproducing Kernel Banach Spaces
Rui Wang, Yuesheng Xu, Mingsong Yan
TL;DR
This work identifies sparsity-promoting reproducing kernel Banach spaces (RKBSs) and derives data-dependent representer theorems for the minimum norm interpolation ($MNI$) and regularization problems. By combining pre-dual driven representer results with two structural assumptions (A1) and (A2), it shows that $MNI$ solutions admit sparse kernel representations whose terms are drawn from a data-dependent set of kernel sessions, with a bound tied to $\mathrm{rank}(\mathbf{L}_{\hat{\nu}})$. The authors specialize the theory to $\ell_1(\mathbb{N})$ and a measure-space RKBS, proving that sparsity can be achieved and can be enhanced by the regularization parameter, while also showing that $\ell_p(\mathbb{N})$ spaces with $1<p<\infty$ do not promote sparsity in general. Additionally, they develop sparsity results for a measure-space RKBS, establishing sparse representations via finite kernel matrices $\mathbf{V}_g$ when a finite support condition (A3) holds. Overall, the results provide practical pathways to obtain compact, data-dependent kernel representations in RKBS-based learning with potential computational benefits for large-scale problems.
Abstract
Sparsity of a learning solution is a desirable feature in machine learning. Certain reproducing kernel Banach spaces (RKBSs) are appropriate hypothesis spaces for sparse learning methods. The goal of this paper is to understand what kind of RKBSs can promote sparsity for learning solutions. We consider two typical learning models in an RKBS: the minimum norm interpolation (MNI) problem and the regularization problem. We first establish an explicit representer theorem for solutions of these problems, which represents the extreme points of the solution set by a linear combination of the extreme points of the subdifferential set, of the norm function, which is data-dependent. We then propose sufficient conditions on the RKBS that can transform the explicit representation of the solutions to a sparse kernel representation having fewer terms than the number of the observed data. Under the proposed sufficient conditions, we investigate the role of the regularization parameter on sparsity of the regularized solutions. We further show that two specific RKBSs: the sequence space $\ell_1(\mathbb{N})$ and the measure space can have sparse representer theorems for both MNI and regularization models.