A Lightweight Randomized Nonlinear Dictionary Learning Method using Random Vector Functional Link
G. Madhuri, Atul Negi
TL;DR
The paper tackles the computational bottleneck of kernel-based nonlinear dictionary learning by proposing an SVD-free, lightweight framework (RVFLDL) that learns a nonlinear dictionary via a Random Vector Functional Link network. Sparse coefficients with Horseshoe priors w.r.t. a random dictionary are nonlinearly transformed and mapped to dense input features, with the dictionary $D_1$ and (optionally) a classifier $W$ obtained in closed form: $D_1 = Y X_1^T (X_1 X_1^T + \mu_1 I)^{-1}$ and $W = H X_1^T (X_1 X_1^T + \mu_2 I)^{-1}$. The method integrates a sparse-to-dense embedding, nonlinear feature augmentation, and analytical solutions to deliver scalable performance across image classification and reconstruction tasks, outperforming several nonlinear DL baselines while avoiding iterative SVD and backpropagation. The approach leverages a HS prior to induce sparsity, reducing model complexity and allowing effective learning with limited data; k-fold averaging further stabilizes the solutions. Empirically, RVFLDL demonstrates strong classification accuracy and image reconstruction quality on diverse datasets (MNIST, USPS, ARDIS, YaleB, COIL100, Fashion-MNIST), suggesting practical impact for high-dimensional, multi-class problems with intra-class variability and limited training data.
Abstract
Kernel-based nonlinear dictionary learning methods operate in a feature space obtained by an implicit feature map, and they are not independent of computationally expensive operations like Singular Value Decomposition (SVD). This paper presents an SVD-free lightweight approach to learning a nonlinear dictionary using a randomized functional link called a Random Vector Functional Link (RVFL). The proposed RVFL-based nonlinear Dictionary Learning (RVFLDL) learns a dictionary as a sparse-to-dense feature map from nonlinear sparse coefficients to the dense input features. Sparse coefficients w.r.t an initial random dictionary are derived by assuming Horseshoe prior are used as inputs making it a lightweight network. Training the RVFL-based dictionary is free from SVD computation as RVFL generates weights from the input to the output layer analytically. Higher-order dependencies between the input sparse coefficients and the dictionary atoms are incorporated into the training process by nonlinearly transforming the sparse coefficients and adding them as enhanced features. Thus the method projects sparse coefficients to a higher dimensional space while inducing nonlinearities into the dictionary. For classification using RVFL-net, a classifier matrix is learned as a transform that maps nonlinear sparse coefficients to the labels. The empirical evidence of the method illustrated in image classification and reconstruction applications shows that RVFLDL is scalable and provides a solution better than those obtained using other nonlinear dictionary learning methods.