Privacy-Preserving Diffusion Model Using Homomorphic Encryption

TL;DR

HE-Diffusion addresses the privacy challenges of running stable diffusion inferences on sensitive data by introducing a practical HE-based framework that protects the denoising step. The approach combines a distortion-based partial encryption strategy with sparse tensor representations to mitigate HE overhead while preserving image quality. Key contributions include a novel HE-enabled inference pipeline for diffusion, a min-distortion partial encryption technique, and COO sparse tensors enabling efficient plaintext-ciphertext hybrid computation. The results demonstrate substantial speedups and near-plaintext accuracy, enabling privacy-preserving diffusion suitable for sensitive applications such as healthcare and confidential design.

Abstract

In this paper, we introduce a privacy-preserving stable diffusion framework leveraging homomorphic encryption, called HE-Diffusion, which primarily focuses on protecting the denoising phase of the diffusion process. HE-Diffusion is a tailored encryption framework specifically designed to align with the unique architecture of stable diffusion, ensuring both privacy and functionality. To address the inherent computational challenges, we propose a novel min-distortion method that enables efficient partial image encryption, significantly reducing the overhead without compromising the model's output quality. Furthermore, we adopt a sparse tensor representation to expedite computational operations, enhancing the overall efficiency of the privacy-preserving diffusion process. We successfully implement HE-based privacy-preserving stable diffusion inference. The experimental results show that HE-Diffusion achieves 500 times speedup compared with the baseline method, and reduces time cost of the homomorphically encrypted inference to the minute level. Both the performance and accuracy of the HE-Diffusion are on par with the plaintext counterpart. Our approach marks a significant step towards integrating advanced cryptographic techniques with state-of-the-art generative models, paving the way for privacy-preserving and efficient image generation in critical applications.

Figures (11)

• Figure 1: Overview of HE-Diffusion at iteration $i$. Users provide embedded prompts tensors, intermediate weights, and the encrypted image $enc\_x_i$. Prompts are embedded locally by users, and the tensor is uploaded to server. The first few layers of the neural network are executed by users. The server performs forward propagation and denoising operations. After that, the server returns the encrypted result, i.e. $enc\_x_{i+1}$, to users.
• Figure 2: When adding distortion in the latent space, the image changes as the figure shows. Specifically, we randomly set elements to 0 in the latent space to generate images with different distortion from the original image. Then, we transform them back to the pixel space. As the distortion grows, from top left to bottom right, the images become noisier and preserve less information.
• Figure 3: Overview of our input protection scheme. Image $x_i$ will not be directly uploaded to the server. The first few layers will be executed by users. Then, the server performs the rest of computation with the conditioning tensor and the intermediate result.
• Figure 4: Overview of our partial encryption scheme. $x_i$ is divided into $y_i$ and $z_i$, with most of information kept by $y_i$. Then $y_i$ is encrypted while $z_i$ remains unencrypted. After denoising, $x_{i+1}$ can be constructed by merging $y_{i+1}$ and $z_{i+1}$.
• Figure 5: Examples to compare the generated images from the plaintext version and the encrypted version. Top row: samples from stable diffusion model. Bottom row: samples from HE-Diffusion model (Prompts: "Night Scene with 2 moons." The height and width of the images are equally and sequentially set as {256, 384, 512, 768, 1024, 1280}. The images corresponding to these size are listed from left to right. All other parameters are equally set).