An introductory Generalization of the standard SVMs loss and its applications to Shallow and Deep Neural Networks

TL;DR

This work introduces a Generalised Loss for SVMs that incorporates pattern correlations through a pattern-correlation matrix $S$, yielding convex dual formulations for both SVC and SVR. The authors derive dual optimization problems, implement WSS-based solvers, and extend the loss to shallow and deep neural networks where the matrix $S$ can be computed per batch or precomputed for small datasets. Across a spectrum of small-scale binary classification, regression, graph, and image tasks, the Generalised Loss matches or improves upon standard losses, though scalability remains a key challenge for large deep networks due to kernel-based computations. The study highlights promising directions for efficient $S$ construction (e.g., anisotropic kernels) and dataset-tailored selection of $S$, with potential impact on generalisation in both shallow and deep learning contexts.

Abstract

We propose a new convex loss for SVMs, both for the binary classification and for the regression models. Therefore, we show the mathematical derivation of the dual problems and we experiment them with several small data-sets. The minimal dimension of those data-sets is due to the difficult scalability of the SVM method to bigger instances. This preliminary study should prove that using pattern correlations inside the loss function could enhance the generalisation performances. Coherently, results show that generalisation measures are never worse than the standard losses and several times they are better. In our opinion, it should be considered a careful study of this loss, coupled with shallow and deep neural networks. In fact, we present some novel results obtained with those architectures.