Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HyPHEN: A Hybrid Packing Method and Optimizations for Homomorphic Encryption-Based Neural Networks

Donghwan Kim, Jaiyoung Park, Jongmin Kim, Sangpyo Kim, Jung Ho Ahn

TL;DR

The paper tackles the practicality gap of HCNN private inference by introducing HyPHEN, a hybrid packing method that reduces memory and rotation/bootstrapping costs. It presents CAConv and RAConv, 2D gap packing, and PRCR as core techniques, with AESPA activations to minimize bootstrapping. The authors demonstrate end-to-end HCNN inference on CIFAR-10 (ResNet-20) at 1.40 seconds and ImageNet (ResNet-18) at 14.69 seconds on GPU, showcasing real-world scalability. The approach also targets memory bottlenecks by reducing plaintext weight sizes and enabling larger models and inputs, enabling private inference at practical latency.

Abstract

Convolutional neural network (CNN) inference using fully homomorphic encryption (FHE) is a promising private inference (PI) solution due to the capability of FHE that enables offloading the whole computation process to the server while protecting the privacy of sensitive user data. Prior FHE-based CNN (HCNN) work has demonstrated the feasibility of constructing deep neural network architectures such as ResNet using FHE. Despite these advancements, HCNN still faces significant challenges in practicality due to the high computational and memory overhead. To overcome these limitations, we present HyPHEN, a deep HCNN construction that incorporates novel convolution algorithms (RAConv and CAConv), data packing methods (2D gap packing and PRCR scheme), and optimization techniques tailored to HCNN construction. Such enhancements enable HyPHEN to substantially reduce the memory footprint and the number of expensive homomorphic operations, such as ciphertext rotation and bootstrapping. As a result, HyPHEN brings the latency of HCNN CIFAR-10 inference down to a practical level at 1.4 seconds (ResNet-20) and demonstrates HCNN ImageNet inference for the first time at 14.7 seconds (ResNet-18).

HyPHEN: A Hybrid Packing Method and Optimizations for Homomorphic Encryption-Based Neural Networks

TL;DR

The paper tackles the practicality gap of HCNN private inference by introducing HyPHEN, a hybrid packing method that reduces memory and rotation/bootstrapping costs. It presents CAConv and RAConv, 2D gap packing, and PRCR as core techniques, with AESPA activations to minimize bootstrapping. The authors demonstrate end-to-end HCNN inference on CIFAR-10 (ResNet-20) at 1.40 seconds and ImageNet (ResNet-18) at 14.69 seconds on GPU, showcasing real-world scalability. The approach also targets memory bottlenecks by reducing plaintext weight sizes and enabling larger models and inputs, enabling private inference at practical latency.

Abstract

Convolutional neural network (CNN) inference using fully homomorphic encryption (FHE) is a promising private inference (PI) solution due to the capability of FHE that enables offloading the whole computation process to the server while protecting the privacy of sensitive user data. Prior FHE-based CNN (HCNN) work has demonstrated the feasibility of constructing deep neural network architectures such as ResNet using FHE. Despite these advancements, HCNN still faces significant challenges in practicality due to the high computational and memory overhead. To overcome these limitations, we present HyPHEN, a deep HCNN construction that incorporates novel convolution algorithms (RAConv and CAConv), data packing methods (2D gap packing and PRCR scheme), and optimization techniques tailored to HCNN construction. Such enhancements enable HyPHEN to substantially reduce the memory footprint and the number of expensive homomorphic operations, such as ciphertext rotation and bootstrapping. As a result, HyPHEN brings the latency of HCNN CIFAR-10 inference down to a practical level at 1.4 seconds (ResNet-20) and demonstrates HCNN ImageNet inference for the first time at 14.7 seconds (ResNet-18).
Paper Structure (4 sections, 4 equations, 3 figures, 1 table, 1 algorithm)

This paper contains 4 sections, 4 equations, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Fully Homomorphic Encryption (FHE)
  4. Convolution on HE

Figures (3)

  • Figure 1: FHE-based private inference.
  • Figure 2: Single-input and single-output channel convolution (SISO)juvekar_2018_gazelle. Image ciphertexts and filter plaintexts are illustrated as 2D matrices, but are stored in a 1D manner with each matrix row concatenated. $\odot$ symbolizes MulPt and $\oplus$ symbolizes AddCt.
  • Figure 3: Convolution procedure in lee_2022_low when $c_i,c_o=2$. A single superscript denotes the channel and a superscript pair denotes (input channel, output channel). $A^{(m)}$ represents the $m$-th channel of the input image contiguously placed in the slots and $K^{(m,n)}$ is the corresponding filter deployed in plaintexts. We simplify the notation of the intermediate SISO result $\sum_{i=1}^{f^2} {CRot(A^{(m)};r(i))K_i^{(m,n)}}$ as ${B}^{(m,n)}$. $C^{(n)}$ represents the $n$-th channel of the output image.