Optimal differentially private kernel learning with random projection

TL;DR

This work tackles optimal differential privacy in kernel ERM by introducing a random-projection scheme in RKHS built from Gaussian processes. The approach (random GP-based projection) enables a finite-dimensional approximation that preserves essential spectral structure, leading to minimax-optimal excess risk bounds for squared and Lipschitz-smooth losses under local strong convexity. The authors show that traditional dimension-reduction methods like random Fourier features are suboptimal in general, derive dimension-free generalization bounds for objective-perturbation private learners, and provide sharper rates for existing DP kernel ERM algorithms. Empirical results on synthetic and real data corroborate theory, demonstrating statistically efficient, optimally private kernel learning and highlighting the pivotal role of dimension reduction in privacy-utility trade-offs.

Abstract

Differential privacy has become a cornerstone in the development of privacy-preserving learning algorithms. This work addresses optimizing differentially private kernel learning within the empirical risk minimization (ERM) framework. We propose a novel differentially private kernel ERM algorithm based on random projection in the reproducing kernel Hilbert space using Gaussian processes. Our method achieves minimax-optimal excess risk for both the squared loss and Lipschitz-smooth convex loss functions under a local strong convexity condition. We further show that existing approaches based on alternative dimension reduction techniques, such as random Fourier feature mappings or $\ell_2$ regularization, yield suboptimal generalization performance. Our key theoretical contribution also includes the derivation of dimension-free generalization bounds for objective perturbation-based private linear ERM -- marking the first such result that does not rely on noisy gradient-based mechanisms. Additionally, we obtain sharper generalization bounds for existing differentially private kernel ERM algorithms. Empirical evaluations support our theoretical claims, demonstrating that random projection enables statistically efficient and optimally private kernel learning. These findings provide new insights into the design of differentially private algorithms and highlight the central role of dimension reduction in balancing privacy and utility.

Figures (12)

• Figure 1: Average test errors for DP kernel ridge regression based on random projection (orange) and RFF (blue) over 100 repetitions. The random projection consistently outperforms RFF across a wide range of privacy budgets, with the performance gap widening as the dimensionality of the original data increases.
• Figure 2: Test errors of DP kernel ridge regression on 20-dimensional synthetic dataset across projection dimensions under different levels of privacy budget $\epsilon$.
• Figure 3: (a) Test errors of DP kernel ridge regression for Million Song dataset. (b) Test errors across projection dimensions $M$ for $\epsilon=10^{-0.5}$ and $\sigma=2^4$.
• Figure 4: Test errors of DP kernel ridge regression on the 10-dimensional synthetic dataset across projection dimensions under different privacy budgets $\epsilon$.
• Figure 5: Test errors of DP kernel ridge regression on the 20-dimensional synthetic dataset across projection dimensions under different privacy budgets $\epsilon$.

Theorems & Definitions (87)

• Definition 1: $(\epsilon,\delta)$-DP, dwork2006our
• Definition 2: Gaussian process
• Definition 3: Random projection on RKHS
• Definition 4: Random projection in Euclidean space, achlioptas2001database
• Theorem 1
• Theorem 2
• Theorem 3
• Theorem 4
• Lemma 1
• Lemma 2