Exploiting Unstructured Sparsity in Fully Homomorphic Encrypted DNNs

TL;DR

This work tackles the high overhead of fully homomorphic encryption for DNN inference by exploiting unstructured sparsity in matrix multiplication. It introduces three sparse FHE matmul schemes (Naïve Sparse, CSR, ELLPACK) and a CPU multi-threading approach within the CKKS framework to accelerate encrypted matrix operations while maintaining accuracy. The results show notable gains, including a $2.5×$ average speedup at 50% sparsity and up to $32.5×$ with 64 cores, along with memory savings and per-element error below $10^{-3}$. These findings enable more practical privacy-preserving DNN inference on commodity hardware and are supported by an open-source implementation.

Abstract

The deployment of deep neural networks (DNNs) in privacy-sensitive environments is constrained by computational overheads in fully homomorphic encryption (FHE). This paper explores unstructured sparsity in FHE matrix multiplication schemes as a means of reducing this burden while maintaining model accuracy requirements. We demonstrate that sparsity can be exploited in arbitrary matrix multiplication, providing runtime benefits compared to a baseline naive algorithm at all sparsity levels. This is a notable departure from the plaintext domain, where there is a trade-off between sparsity and the overhead of the sparse multiplication algorithm. In addition, we propose three sparse multiplication schemes in FHE based on common plaintext sparse encodings. We demonstrate the performance gain is scheme-invariant; however, some sparse schemes vastly reduce the memory storage requirements of the encrypted matrix at high sparsity values. Our proposed sparse schemes yield an average performance gain of 2.5x at 50% unstructured sparsity, with our multi-threading scheme providing a 32.5x performance increase over the equivalent single-threaded sparse computation when utilizing 64 cores.

Figures (8)

• Figure 1: Execution time of square matrix multiplication in plaintext and FHE. Note that the Y-Axis is logarithmic.
• Figure 2: Runtime of dense and sparse schemes multiplying two $8 \times 8$ matrices. Shaded region denotes the runtime advantage of utilizing sparsity.
• Figure 3: ELLPACK sparse scheme multi-threading matmul runtime. Multiplying $8 \times 8$ matrices with chunk size $c = 1$.
• Figure 4: ELLPACK sparse scheme ciphertext memory usage. Multiplying $8 \times 8$ matrices with chunk size $c = 1$. The lines represent different thread count levels used for the computation with the shaded regions denoting $1 \sigma$ deviation, highlighting the low memory overhead of our multi-threading scheme relative to the ciphertext size.
• Figure 5: Relative runtime (HEMat excluded due to high runtime) between schemes when multiplying $8 \times 8$ matrices with 16 threads.