FHEON: A Configurable Framework for Developing Privacy-Preserving Neural Networks Using Homomorphic Encryption

TL;DR

FHEON addresses privacy concerns in MLaaS by enabling encrypted inference for CNNs using CKKS, providing a configurable, open-source framework of HE-friendly layers (convolution, pooling, ReLU, and fully connected) and system-level optimizations. It demonstrates near-plaintext accuracy across multiple architectures (LeNet-5, VGG-11/16, ResNet-20/34) on standard datasets while running on consumer-grade hardware, attaining competitive latencies and memory footprints (e.g., 13s for MNIST LeNet-5, 403s for CIFAR-10 ResNet-20, and sub-50 GB footprints across models). The contributions include configurable HE layers, evaluation-key reuse strategies, flexible weight loading, and bootstrapping integration, all enabling broader applicability of encrypted CNN inference. This work significantly lowers the barrier to practical adoption of privacy-preserving ML in real-world MLaaS pipelines by balancing accuracy, performance, and memory on accessible hardware, and it provides a foundation for extending to encrypted training and additional NN architectures.

Abstract

The widespread adoption of Machine Learning as a Service raises critical privacy and security concerns, particularly about data confidentiality and trust in both cloud providers and the machine learning models. Homomorphic Encryption (HE) has emerged as a promising solution to this problems, allowing computations on encrypted data without decryption. Despite its potential, existing approaches to integrate HE into neural networks are often limited to specific architectures, leaving a wide gap in providing a framework for easy development of HE-friendly privacy-preserving neural network models similar to what we have in the broader field of machine learning. In this paper, we present FHEON, a configurable framework for developing privacy-preserving convolutional neural network (CNN) models for inference using HE. FHEON introduces optimized and configurable implementations of privacy-preserving CNN layers including convolutional layers, average pooling layers, ReLU activation functions, and fully connected layers. These layers are configured using parameters like input channels, output channels, kernel size, stride, and padding to support arbitrary CNN architectures. We assess the performance of FHEON using several CNN architectures, including LeNet-5, VGG-11, VGG- 16, ResNet-20, and ResNet-34. FHEON maintains encrypted-domain accuracies within +/- 1% of their plaintext counterparts for ResNet-20 and LeNet-5 models. Notably, on a consumer-grade CPU, the models build on FHEON achieved 98.5% accuracy with a latency of 13 seconds on MNIST using LeNet-5, and 92.2% accuracy with a latency of 403 seconds on CIFAR-10 using ResNet-20. Additionally, FHEON operates within a practical memory budget requiring not more than 42.3 GB for VGG-16.

Figures (6)

• Figure 1: ML inference in a cloud scenario showing a user inferring encrypted data over an encrypted model.
• Figure 2: Flattening of an Input tensor into a SIMD Vector to be encrypted and used as an input of FHEON
• Figure 3: Rotated SIMD ciphertexts multiplied with their equivalent kernel vectors and summed to produce the $A$ ciphertext.
• Figure 4: Secure convolution using SIMD showing the extraction process and addition of bias
• Figure 5: Secure Average Pooling with kernel scaling value of $\frac{1}{k^2}$. The resulting ciphertext is passed to a striding module.