Multi-Class Support Vector Machine with Differential Privacy

TL;DR

This work tackles the challenge of enforcing differential privacy in multi-class SVMs by introducing PMSVM, an all-in-one framework that reduces data accesses and hence privacy budget consumption. It presents two DP perturbation schemes: weight perturbation (WP) and gradient perturbation (GP), with formal DP guarantees and convergence analyses, leveraging a Gram-matrix–based sensitivity bound and a smoothed loss for DP gradient updates. Empirically, PMSVM outperforms existing DP-SVM baselines across several multi-class datasets, especially under stringent privacy budgets, illustrating a favorable privacy-utility trade-off. The results suggest that single-access, all-in-one DP-SVMs are practical for privacy-preserving multi-class classification, potentially enabling broader deployment in privacy-conscious applications.

Abstract

With the increasing need to safeguard data privacy in machine learning models, differential privacy (DP) is one of the major frameworks to build privacy-preserving models. Support Vector Machines (SVMs) are widely used traditional machine learning models due to their robust margin guarantees and strong empirical performance in binary classification. However, applying DP to multi-class SVMs is inadequate, as the standard one-versus-rest (OvR) and one-versus-one (OvO) approaches repeatedly query each data sample when building multiple binary classifiers, thus consuming the privacy budget proportionally to the number of classes. To overcome this limitation, we explore all-in-one SVM approaches for DP, which access each data sample only once to construct multi-class SVM boundaries with margin maximization properties. We propose a novel differentially Private Multi-class SVM (PMSVM) with weight and gradient perturbation methods, providing rigorous sensitivity and convergence analyses to ensure DP in all-in-one SVMs. Empirical results demonstrate that our approach surpasses existing DP-SVM methods in multi-class scenarios.

Figures (6)

• Figure 1: Illustration of multi-class classification strategies for $c$ classes. The individual sample ($\star$) is queried repeatedly in (a) and (b), but only once in (c). Each color represents a class.
• Figure 2: Accuracy gap between DP-SVM methods and their non-private baselines ($\epsilon=\infty$). Lower value indicates a smaller accuracy–privacy trade-off, thus indicating a DP-friendly property.
• Figure 3: Convergence curves of training loss, training accuracy, and test accuracy for the proposed PMSVM-GP and PMSVM-AGP methods.
• Figure 4: Weight Perturbation; Accuracy gap between DP-SVM methods and their non-private baselines ($\epsilon=\infty$). Lower value indicates a smaller accuracy–privacy trade-off, thus indicating a DP-friendly property.
• Figure 5: Gradient Perturbation; Accuracy gap between DP-SVM methods and their non-private baselines ($\epsilon=\infty$). Lower value indicates a smaller accuracy–privacy trade-off, thus indicating a DP-friendly property.

Theorems & Definitions (16)

• Definition 1
• Definition 2: $L_2$ Sensitivity
• Remark 1
• Remark 2
• Remark 3
• Definition 3: Weight Perturbation
• Lemma 1
• Theorem 1: DP guarantee of weight perturbation
• Theorem 1: DP guarantee of weight perturbation
• Definition 4: Gradient Perturbation