Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Unified Optimization Framework for Multiclass Classification with Structured Hyperplane Arrangements

Víctor Blanco, Harshit Kothari, James Luedtke

TL;DR

The paper addresses multiclass classification by proposing a unified optimization framework based on structured hyperplane arrangements, formulated as a MIP that generalizes the SVM margin principle to multiple hyperplanes. It introduces a computationally more efficient New Formulation that reduces binary variables to $(n + k)|\mathcal{C}|$ with $|\mathcal{C}| = 2^m$, and extends to decision-tree structures and kernelized nonlinear boundaries via a kernel trick. A dynamic clustering matheuristic is developed to scale to large datasets, and extensive experiments on synthetic and UCI datasets show competitive accuracy with substantial computational gains over prior discrete optimization approaches. The framework offers interpretable, margin-based multiclass classifiers and provides avenues for robustness and decomposition-based scalability improvements.

Abstract

In this paper, we propose a new mathematical optimization model for multiclass classification based on arrangements of hyperplanes. Our approach preserves the core support vector machine (SVM) paradigm of maximizing class separation while minimizing misclassification errors, and it is computationally more efficient than a previous formulation. We present a kernel-based extension that allows it to construct nonlinear decision boundaries. Furthermore, we show how the framework can naturally incorporate alternative geometric structures, including classification trees, $\ell_p$-SVMs, and models with discrete feature selection. To address large-scale instances, we develop a dynamic clustering matheuristic that leverages the proposed MIP formulation. Extensive computational experiments demonstrate the efficiency of the proposed model and dynamic clustering heuristic, and we report competitive classification performance on both synthetic datasets and real-world benchmarks from the UCI Machine Learning Repository, comparing our method with state-of-the-art implementations available in scikit-learn.

A Unified Optimization Framework for Multiclass Classification with Structured Hyperplane Arrangements

TL;DR

The paper addresses multiclass classification by proposing a unified optimization framework based on structured hyperplane arrangements, formulated as a MIP that generalizes the SVM margin principle to multiple hyperplanes. It introduces a computationally more efficient New Formulation that reduces binary variables to with , and extends to decision-tree structures and kernelized nonlinear boundaries via a kernel trick. A dynamic clustering matheuristic is developed to scale to large datasets, and extensive experiments on synthetic and UCI datasets show competitive accuracy with substantial computational gains over prior discrete optimization approaches. The framework offers interpretable, margin-based multiclass classifiers and provides avenues for robustness and decomposition-based scalability improvements.

Abstract

In this paper, we propose a new mathematical optimization model for multiclass classification based on arrangements of hyperplanes. Our approach preserves the core support vector machine (SVM) paradigm of maximizing class separation while minimizing misclassification errors, and it is computationally more efficient than a previous formulation. We present a kernel-based extension that allows it to construct nonlinear decision boundaries. Furthermore, we show how the framework can naturally incorporate alternative geometric structures, including classification trees, -SVMs, and models with discrete feature selection. To address large-scale instances, we develop a dynamic clustering matheuristic that leverages the proposed MIP formulation. Extensive computational experiments demonstrate the efficiency of the proposed model and dynamic clustering heuristic, and we report competitive classification performance on both synthetic datasets and real-world benchmarks from the UCI Machine Learning Repository, comparing our method with state-of-the-art implementations available in scikit-learn.

Paper Structure

This paper contains 14 sections, 4 theorems, 19 equations, 9 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Multiclass Classification with Structured Hyperplane Arrangements
  3. Mathematical Optimization Models
  4. Prior Formulation
  5. New Formulation
  6. Extension to Classification Via Decision Trees
  7. A Kernelized Version
  8. Computational Experiments
  9. Synthetic Datasets
  10. Computational Performance
  11. Prediction Accuracy
  12. Scaling Up Via Dynamic Clustering
  13. Performance on Real Datasets
  14. Conclusion

Key Result

Proposition 1

Integrality constraints tu:binary can be relaxed.

Figures (9)

  • Figure 1: Dataset to illustrate the methodology.
  • Figure 2: Solutions obtained with our methodology (left), classical linear SVM-based methods (center), and rbf SVM-based methods .
  • Figure 3: Solution obtained with the kernelized model for Example \ref{['ex:2']}
  • Figure 4: Illustration of one of our 10-dimensional instances (projected onto the two first dimensions) with 3 classes with three clusters for each of them for a sample to size $1000$.
  • Figure 5: Performance Profile by size of the datasets by the breaking symmetry strategy used. CPU time is in log-scale.
  • ...and 4 more figures

Theorems & Definitions (10)

  • Example 1
  • Proposition 1
  • proof
  • Proposition 2
  • proof
  • Theorem 3
  • proof
  • Proposition 4
  • proof
  • Example 2