Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Privacy-Preserving CNN Training with Transfer Learning: Two Hidden Layers

John Chiang

TL;DR

This work tackles the problem of training CNNs on encrypted data using fully homomorphic encryption (FHE) in a non-interactive setting. The authors identify Binary Cross-Entropy (BCE) as a stable objective that extends naturally to multi-output and multi-class tasks, and they introduce the Double Volley Revolver encoding along with ciphertext packing to improve scalability and parallelism. By freezing a pre-trained REGNET-X_400MF feature extractor and training only the final layer via BCE, they demonstrate transfer-learning-based encrypted training on USPS, MNIST, and CIFAR-10 with two iterations, highlighting practical trade-offs and limitations, such as bootstrapping constraints. The work advances practical privacy-preserving training by combining polynomial-approximation techniques for nonlinearities with efficient encrypted matrix operations, and provides runnable C++ code to facilitate further research, albeit with room for optimization in batching and bootstrapping strategies ($λ=128$ security).

Abstract

In this paper, we present the demonstration of training a four-layer neural network entirely using fully homomorphic encryption (FHE), supporting both single-output and multi-output classification tasks in a non-interactive setting. A key contribution of our work is identifying that replacing \textit{Softmax} with \textit{Sigmoid}, in conjunction with the Binary Cross-Entropy (BCE) loss function, provides an effective and scalable solution for homomorphic classification. Moreover, we show that the BCE loss function, originally designed for multi-output tasks, naturally extends to the multi-class setting, thereby enabling broader applicability. We also highlight the limitations of prior loss functions such as the SLE loss and the one proposed in the 2019 CVPR Workshop, both of which suffer from vanishing gradients as network depth increases. To address the challenges posed by large-scale encrypted data, we further introduce an improved version of the previously proposed data encoding scheme, \textit{Double Volley Revolver}, which achieves a better trade-off between computational and memory efficiency, making FHE-based neural network training more practical. The complete, runnable C++ code to implement our work can be found at: \href{https://github.com/petitioner/ML.NNtraining}{$\texttt{https://github.com/petitioner/ML.NNtraining}$}.

Privacy-Preserving CNN Training with Transfer Learning: Two Hidden Layers

TL;DR

This work tackles the problem of training CNNs on encrypted data using fully homomorphic encryption (FHE) in a non-interactive setting. The authors identify Binary Cross-Entropy (BCE) as a stable objective that extends naturally to multi-output and multi-class tasks, and they introduce the Double Volley Revolver encoding along with ciphertext packing to improve scalability and parallelism. By freezing a pre-trained REGNET-X_400MF feature extractor and training only the final layer via BCE, they demonstrate transfer-learning-based encrypted training on USPS, MNIST, and CIFAR-10 with two iterations, highlighting practical trade-offs and limitations, such as bootstrapping constraints. The work advances practical privacy-preserving training by combining polynomial-approximation techniques for nonlinearities with efficient encrypted matrix operations, and provides runnable C++ code to facilitate further research, albeit with room for optimization in batching and bootstrapping strategies ( security).

Abstract

In this paper, we present the demonstration of training a four-layer neural network entirely using fully homomorphic encryption (FHE), supporting both single-output and multi-output classification tasks in a non-interactive setting. A key contribution of our work is identifying that replacing \textit{Softmax} with \textit{Sigmoid}, in conjunction with the Binary Cross-Entropy (BCE) loss function, provides an effective and scalable solution for homomorphic classification. Moreover, we show that the BCE loss function, originally designed for multi-output tasks, naturally extends to the multi-class setting, thereby enabling broader applicability. We also highlight the limitations of prior loss functions such as the SLE loss and the one proposed in the 2019 CVPR Workshop, both of which suffer from vanishing gradients as network depth increases. To address the challenges posed by large-scale encrypted data, we further introduce an improved version of the previously proposed data encoding scheme, \textit{Double Volley Revolver}, which achieves a better trade-off between computational and memory efficiency, making FHE-based neural network training more practical. The complete, runnable C++ code to implement our work can be found at: \href{https://github.com/petitioner/ML.NNtraining}{}.

Paper Structure

This paper contains 30 sections, 15 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Related Work
  4. Contributions
  5. Preliminaries
  6. Fully Homomorphic Encryption
  7. Data Encoding
  8. Convolutional Neural Network
  9. Transfer Learning
  10. REGNET_X_400MF
  11. Squared Likelihood-Error Loss
  12. Technical Details
  13. Binary Cross-Entropy Loss
  14. Double Volley Revolver
  15. Vertical Partitioning
  16. ...and 15 more sections

Figures (2)

  • Figure 1: The matrix multiplication algorithm of $\texttt{Volley Revolver}$ for the $4 \times 2$ matrix $A$ and the matrix $B$ of size $2 \times 2$
  • Figure 2: A typical neural network with two hidden layers is illustrated, where black circles represent bias nodes that constantly emit a value of 1. The weight matrices $W^{(\ell)}$ determine the contribution of each input signal to the activation function at each node.