Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diversifying Support Vector Machines for Boosting using Kernel Perturbation: Applications to Class Imbalance and Small Disjuncts

Shounak Datta, Sayak Nag, Sankha Subhra Mullick, Swagatam Das

TL;DR

This work proposes two kernel perturbation based boosting schemes where the kernel is modified in each round so as to increase the resolution of the kernel-induced Reimannian metric in the vicinity of the datapoints misclassified in the previous round.

Abstract

The diversification (generating slightly varying separating discriminators) of Support Vector Machines (SVMs) for boosting has proven to be a challenge due to the strong learning nature of SVMs. Based on the insight that perturbing the SVM kernel may help in diversifying SVMs, we propose two kernel perturbation based boosting schemes where the kernel is modified in each round so as to increase the resolution of the kernel-induced Reimannian metric in the vicinity of the datapoints misclassified in the previous round. We propose a method for identifying the disjuncts in a dataset, dispelling the dependence on rule-based learning methods for identifying the disjuncts. We also present a new performance measure called Geometric Small Disjunct Index (GSDI) to quantify the performance on small disjuncts for balanced as well as class imbalanced datasets. Experimental comparison with a variety of state-of-the-art algorithms is carried out using the best classifiers of each type selected by a new approach inspired by multi-criteria decision making. The proposed method is found to outperform the contending state-of-the-art methods on different datasets (ranging from mildly imbalanced to highly imbalanced and characterized by varying number of disjuncts) in terms of three different performance indices (including the proposed GSDI).

Diversifying Support Vector Machines for Boosting using Kernel Perturbation: Applications to Class Imbalance and Small Disjuncts

TL;DR

This work proposes two kernel perturbation based boosting schemes where the kernel is modified in each round so as to increase the resolution of the kernel-induced Reimannian metric in the vicinity of the datapoints misclassified in the previous round.

Abstract

The diversification (generating slightly varying separating discriminators) of Support Vector Machines (SVMs) for boosting has proven to be a challenge due to the strong learning nature of SVMs. Based on the insight that perturbing the SVM kernel may help in diversifying SVMs, we propose two kernel perturbation based boosting schemes where the kernel is modified in each round so as to increase the resolution of the kernel-induced Reimannian metric in the vicinity of the datapoints misclassified in the previous round. We propose a method for identifying the disjuncts in a dataset, dispelling the dependence on rule-based learning methods for identifying the disjuncts. We also present a new performance measure called Geometric Small Disjunct Index (GSDI) to quantify the performance on small disjuncts for balanced as well as class imbalanced datasets. Experimental comparison with a variety of state-of-the-art algorithms is carried out using the best classifiers of each type selected by a new approach inspired by multi-criteria decision making. The proposed method is found to outperform the contending state-of-the-art methods on different datasets (ranging from mildly imbalanced to highly imbalanced and characterized by varying number of disjuncts) in terms of three different performance indices (including the proposed GSDI).

Paper Structure

This paper contains 33 sections, 8 theorems, 31 equations, 2 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Motivation
  3. Contribution
  4. Background on Selective Kernel Perturbation
  5. Kernel Perturbation based Boosting of SVMs
  6. Choice of weights for the component classifiers
  7. Classifying a test point
  8. The complete KPBoost-SVM algorithm
  9. Experiments
  10. Indices for evaluation of classification performance
  11. Gmean
  12. AUC
  13. GSDI
  14. Description of the datasets
  15. Contending algorithms and parameter settings
  16. ...and 18 more sections

Key Result

Theorem 1

For a datapoint $\mathbf{x}_i \in P$ close to the decision boundary $f_t(\mathbf{x}) = 0$ at the $t$-th round we have $m'_{t+1}(\mathbf{x}_i) > m_{t+1}(\mathbf{x}_i)$, where $m'_{t+1}(\mathbf{x}_i)$ is the ratio of the new resolution at the $(t+1)$-th round and the old resolution at the $t$-th round

Figures (2)

  • Figure 1: $\kappa-\delta$ curves for Iris:12vs3 and Balance datasets with the knee-points.
  • Figure 2: Effect of ROI on the estimation of resolution for test points.

Theorems & Definitions (8)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4