RQP-SGD: Differential Private Machine Learning through Noisy SGD and Randomized Quantization
Ce Feng, Parv Venkitasubramaniam
TL;DR
This paper addresses privacy-preserving ML at the edge where model weights must be quantized. It introduces RQP-SGD, a randomized quantization projection-based SGD that injects Gaussian DP noise and uses randomized projection to satisfy privacy while reducing projection error. The authors provide a convergence-utility analysis for convex, Lipschitz losses, showing the excess empirical loss is bounded by terms including quantization error and noise error, with the quantization error depending on bit-depth and randomness. Empirical results on MNIST and a Diagnostic breast cancer dataset demonstrate that RQP-SGD can outperform deterministic-Projection DP-SGD, achieving substantial median accuracy gains under the same privacy budgets, highlighting its practicality for IoT deployment.
Abstract
The rise of IoT devices has prompted the demand for deploying machine learning at-the-edge with real-time, efficient, and secure data processing. In this context, implementing machine learning (ML) models with real-valued weight parameters can prove to be impractical particularly for large models, and there is a need to train models with quantized discrete weights. At the same time, these low-dimensional models also need to preserve privacy of the underlying dataset. In this work, we present RQP-SGD, a new approach for privacy-preserving quantization to train machine learning models for low-memory ML-at-the-edge. This approach combines differentially private stochastic gradient descent (DP-SGD) with randomized quantization, providing a measurable privacy guarantee in machine learning. In particular, we study the utility convergence of implementing RQP-SGD on ML tasks with convex objectives and quantization constraints and demonstrate its efficacy over deterministic quantization. Through experiments conducted on two datasets, we show the practical effectiveness of RQP-SGD.