Support Vector Boosting Machine (SVBM): Enhancing Classification Performance with AdaBoost and Residual Connections
Junbo Jacob Lian
TL;DR
The paper addresses improving classification by boosting SVMs without sacrificing robustness. It introduces the Support Vector Boosting Machine (SVBM), which combines structured subsampling and residual connections to update sample weights using both current predictions and historical information, enabling richer decision boundaries and sparsity control. Validation across ten public datasets shows that SVBM outperforms existing boosted-SVM approaches, supported by ablation studies, and the authors provide open-source MATLAB code for broad accessibility. This approach offers a practical, flexible boosting framework that leverages SVMs more effectively in ensemble settings.
Abstract
In traditional boosting algorithms, the focus on misclassified training samples emphasizes their importance based on difficulty during the learning process. While using a standard Support Vector Machine (SVM) as a weak learner in an AdaBoost framework can enhance model performance by concentrating on error samples, this approach introduces significant challenges. Specifically, SVMs, characterized by their stability and robustness, may require destabilization to fit the boosting paradigm, which in turn can constrain performance due to reliance on the weighted results from preceding iterations. To address these challenges, we propose the Support Vector Boosting Machine (SVBM), which integrates a novel subsampling process with SVM algorithms and residual connection techniques. This method updates sample weights by considering both the current model's predictions and the outputs from prior rounds, allowing for effective sparsity control. The SVBM framework enhances the ability to form complex decision boundaries, thereby improving classification performance. The MATLAB source code for SVBM can be accessed at https://github.com/junbolian/SVBM.