Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Privacy-Preserving Logistic Regression Training with A Faster Gradient Variant

John Chiang

TL;DR

The paper tackles privacy-preserving training of logistic regression on encrypted data by introducing a quadratic gradient that blends first-order gradient information with a diagonal Hessian surrogate, enabling faster convergence. It develops enhanced Nesterov Accelerated Gradient (NAG), Adagrad, and Adam variants that use the quadratic gradient, and demonstrates their effectiveness both in plaintext and in secure, homomorphically encrypted settings (via HEAAN/SIMD). Empirically, the enhanced methods converge faster than their vanilla counterparts, and secure LR training on encrypted data achieves competitive accuracy/AUC with only a few iterations thanks to a degree-5 sigmoid approximation and diagonal Hessian tricks. Overall, the approach provides a practical pathway to accelerate privacy-preserving optimization and could generalize to integrating first- and second-order methods across a broad range of numerical problems.

Abstract

Training logistic regression over encrypted data has been a compelling approach in addressing security concerns for several years. In this paper, we introduce an efficient gradient variant, called $quadratic$ $gradient$, which can be used for privacy-preserving logistic regression training. We enhance Nesterov's Accelerated Gradient (NAG), Adaptive Gradient Algorithm (Adagrad) and Adam algorithms by incorporating their quadratic gradients and evaluate these improved algorithms on various datasets. Experimental results demonstrate that the enhanced algorithms achieve significantly improved convergence speed compared to traditional first-order gradient methods. Moreover, we applied the enhanced NAG method to implement homomorphic logistic regression training, achieving comparable results within just 4 iterations. There is a great chance that the quadratic gradient approach could integrate first-order gradient descent/ascent algorithms with the second-order Newton-Raphson methods, and that quadratic gradient could be applied to a wide range of numerical optimization problems.

Privacy-Preserving Logistic Regression Training with A Faster Gradient Variant

TL;DR

The paper tackles privacy-preserving training of logistic regression on encrypted data by introducing a quadratic gradient that blends first-order gradient information with a diagonal Hessian surrogate, enabling faster convergence. It develops enhanced Nesterov Accelerated Gradient (NAG), Adagrad, and Adam variants that use the quadratic gradient, and demonstrates their effectiveness both in plaintext and in secure, homomorphically encrypted settings (via HEAAN/SIMD). Empirically, the enhanced methods converge faster than their vanilla counterparts, and secure LR training on encrypted data achieves competitive accuracy/AUC with only a few iterations thanks to a degree-5 sigmoid approximation and diagonal Hessian tricks. Overall, the approach provides a practical pathway to accelerate privacy-preserving optimization and could generalize to integrating first- and second-order methods across a broad range of numerical problems.

Abstract

Training logistic regression over encrypted data has been a compelling approach in addressing security concerns for several years. In this paper, we introduce an efficient gradient variant, called , which can be used for privacy-preserving logistic regression training. We enhance Nesterov's Accelerated Gradient (NAG), Adaptive Gradient Algorithm (Adagrad) and Adam algorithms by incorporating their quadratic gradients and evaluate these improved algorithms on various datasets. Experimental results demonstrate that the enhanced algorithms achieve significantly improved convergence speed compared to traditional first-order gradient methods. Moreover, we applied the enhanced NAG method to implement homomorphic logistic regression training, achieving comparable results within just 4 iterations. There is a great chance that the quadratic gradient approach could integrate first-order gradient descent/ascent algorithms with the second-order Newton-Raphson methods, and that quadratic gradient could be applied to a wide range of numerical optimization problems.
Paper Structure (13 sections, 1 theorem, 10 equations, 4 figures, 2 tables, 1 algorithm)

This paper contains 13 sections, 1 theorem, 10 equations, 4 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Fully Homomorphic Encryption
  4. Database Encoding Method
  5. Logistic Regression Model
  6. Technical Details
  7. Simplified Fixed Hessian
  8. Quadratic Gradient: Definition
  9. Quadratic Gradient: Algorithms
  10. Secure Training
  11. Limitations
  12. Experiments
  13. Conclusion

Key Result

Lemma 3.1

Let $A \in \mathbb R^{n \times n}$ be a symmetric matrix, and let $B$ be the diagonal matrix whose diagonal entries $B_{kk} = - \epsilon - \sum_{i} | A_{ki} |$ for $k = 1, \ldots , n$, then $B \le A$.

Figures (4)

  • Figure 1: The training results of NAG vs. Enhanced NAG in the clear domain.
  • Figure 2: The training results of Adagrad vs. Enhanced Adagrad in the clear domain.
  • Figure 3: The training results of Adam vs. Enhanced Adam in the clear domain.
  • Figure 4: Training results of standard first-order and quadratic gradient ascent algorithms on the lbw dataset in the plaintext domain.

Theorems & Definitions (3)

  • Lemma 3.1
  • proof
  • Definition 3.2: $\texttt{Quadratic Gradient}$