Batch-oriented Element-wise Approximate Activation for Privacy-Preserving Neural Networks

TL;DR

This paper tackles privacy-preserving neural networks by employing FHE with RNS-CKKS to perform encrypted inference. It introduces BEAA, a batch-oriented, element-wise activation strategy that trains per-element quadratic polynomials to approximate ReLU, enabling large-batch data packing and improved slot utilization. The method is enhanced with knowledge distillation to further boost accuracy, achieving a 1.65% accuracy gain over channel-wise methods and dramatically lower amortized per-image time as batch size grows. Overall, BEAA offers a practical path to higher-accuracy, scalable PPNN inference on encrypted data for applications such as medical diagnosis.

Abstract

Privacy-Preserving Neural Networks (PPNN) are advanced to perform inference without breaching user privacy, which can serve as an essential tool for medical diagnosis to simultaneously achieve big data utility and privacy protection. As one of the key techniques to enable PPNN, Fully Homomorphic Encryption (FHE) is facing a great challenge that homomorphic operations cannot be easily adapted for non-linear activation calculations. In this paper, batch-oriented element-wise data packing and approximate activation are proposed, which train linear low-degree polynomials to approximate the non-linear activation function - ReLU. Compared with other approximate activation methods, the proposed fine-grained, trainable approximation scheme can effectively reduce the accuracy loss caused by approximation errors. Meanwhile, due to element-wise data packing, a large batch of images can be packed and inferred concurrently, leading to a much higher utility ratio of ciphertext slots. Therefore, although the total inference time increases sharply, the amortized time for each image actually decreases, especially when the batch size increases. Furthermore, knowledge distillation is adopted in the training process to further enhance the inference accuracy. Experiment results show that when ciphertext inference is performed on 4096 input images, compared with the current most efficient channel-wise method, the inference accuracy is improved by 1.65%, and the amortized inference time is reduced by 99.5%.

Figures (7)

• Figure 1: ReLU activation and Layer-wise approximate activation.
• Figure 2: Layer-wise and Channel-wise polynomial approximation.
• Figure 3: Element-wise data packing.
• Figure 4: Element-wise approximate activation.
• Figure 5: The model training with knowledge distillation.