Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PrivCirNet: Efficient Private Inference via Block Circulant Transformation

Tianshi Xu, Lemeng Wu, Runsheng Wang, Meng Li

TL;DR

This paper proposes PrivCirNet, a protocol/network co-optimization framework based on block circulant transformation that customizes the HE encoding algorithm that is fully compatible with the block circulant transformation and reduces the computation latency in proportion to the block size.

Abstract

Homomorphic encryption (HE)-based deep neural network (DNN) inference protects data and model privacy but suffers from significant computation overhead. We observe transforming the DNN weights into circulant matrices converts general matrix-vector multiplications into HE-friendly 1-dimensional convolutions, drastically reducing the HE computation cost. Hence, in this paper, we propose \method, a protocol/network co-optimization framework based on block circulant transformation. At the protocol level, PrivCirNet customizes the HE encoding algorithm that is fully compatible with the block circulant transformation and reduces the computation latency in proportion to the block size. At the network level, we propose a latency-aware formulation to search for the layer-wise block size assignment based on second-order information. PrivCirNet also leverages layer fusion to further reduce the inference cost. We compare PrivCirNet with the state-of-the-art HE-based framework Bolt (IEEE S\&P 2024) and the HE-friendly pruning method SpENCNN (ICML 2023). For ResNet-18 and Vision Transformer (ViT) on Tiny ImageNet, PrivCirNet reduces latency by $5.0\times$ and $1.3\times$ with iso-accuracy over Bolt, respectively, and improves accuracy by $4.1\%$ and $12\%$ over SpENCNN, respectively. For MobileNetV2 on ImageNet, PrivCirNet achieves $1.7\times$ lower latency and $4.2\%$ better accuracy over Bolt and SpENCNN, respectively. Our code and checkpoints are available on Git Hub.

PrivCirNet: Efficient Private Inference via Block Circulant Transformation

TL;DR

This paper proposes PrivCirNet, a protocol/network co-optimization framework based on block circulant transformation that customizes the HE encoding algorithm that is fully compatible with the block circulant transformation and reduces the computation latency in proportion to the block size.

Abstract

Homomorphic encryption (HE)-based deep neural network (DNN) inference protects data and model privacy but suffers from significant computation overhead. We observe transforming the DNN weights into circulant matrices converts general matrix-vector multiplications into HE-friendly 1-dimensional convolutions, drastically reducing the HE computation cost. Hence, in this paper, we propose \method, a protocol/network co-optimization framework based on block circulant transformation. At the protocol level, PrivCirNet customizes the HE encoding algorithm that is fully compatible with the block circulant transformation and reduces the computation latency in proportion to the block size. At the network level, we propose a latency-aware formulation to search for the layer-wise block size assignment based on second-order information. PrivCirNet also leverages layer fusion to further reduce the inference cost. We compare PrivCirNet with the state-of-the-art HE-based framework Bolt (IEEE S\&P 2024) and the HE-friendly pruning method SpENCNN (ICML 2023). For ResNet-18 and Vision Transformer (ViT) on Tiny ImageNet, PrivCirNet reduces latency by and with iso-accuracy over Bolt, respectively, and improves accuracy by and over SpENCNN, respectively. For MobileNetV2 on ImageNet, PrivCirNet achieves lower latency and better accuracy over Bolt and SpENCNN, respectively. Our code and checkpoints are available on Git Hub.
Paper Structure (40 sections, 4 theorems, 23 equations, 19 figures, 9 tables, 1 algorithm)

This paper contains 40 sections, 4 theorems, 23 equations, 19 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Cryptographic Primitives
  4. Threat Model and Security Guarantee
  5. Related Works
  6. PrivCirNet Framework
  7. Motivation
  8. CirEncode: nested encoding for block circulant GEMMs
  9. Latency-aware block size assignment with loss-aware initialization
  10. Network-Protocol Co-Fusion
  11. Experiments
  12. Experimental Setup
  13. Micro-Benchmark on Single GEMM and Convolution
  14. End-to-End Inference Evaluation
  15. Ablation Study
  16. ...and 25 more sections

Key Result

Lemma 1

$\left \langle \operatorname{DFT}(w) \right \rangle_{\mathrm{SIMD}} \times \left \langle \operatorname{DFT}(x) \right \rangle_{\mathrm{SIMD}} = \left \langle \operatorname{DFT}(w)\odot \operatorname{DFT}(x)\right \rangle_{\mathrm{SIMD}}=\operatorname{DFT}(\left \langle w \right \rangle_{\mathrm{Coef

Figures (19)

  • Figure 1: (a) Illustration of Hybrid HE/MPC-based private inference; (b) latency breakdown of linear layers and nonlinear layers based on Bolt's protocol; (c) latency breakdown of linear layers of the original model and SpENCNN with 50% sparsity; (d) GEMV with a circulant weight matrix.
  • Figure 2: Directly using coefficient or SIMD encoding to block circulant GEMMs ($(d_1,d_2,d_3,b)=(256,192,576,2)$) leads to limited efficiency improvement.
  • Figure 3: Overview of PrivCirNet.
  • Figure 4: An example of CirEncode for block circulant GEMM where $(d_1,d_2,d_3,b)=(4,8,8,4)$.
  • Figure 5: Layer-wise sensitivity and block size visualization for ViT on CIFAR-100.
  • ...and 14 more figures

Theorems & Definitions (4)

  • Lemma 1
  • Theorem 1
  • Theorem 2
  • Theorem 3