Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CipherDM: Secure Three-Party Inference for Diffusion Model Sampling

Xin Zhao, Xiaojun Chen, Xudong Chen, He Li, Tingyu Fan, Zhendong Zhao

TL;DR

CipherDM presents the first MPC-based framework for secure diffusion-model sampling using a 3-party ABY3 replicated secret-sharing scheme in an outsourced setting. It identifies nonlinear activations as the main latency bottlenecks and tackles them with secure polynomial protocols for SoftMax and SiLU/Mish, complemented by Chebyshev-based exponentiation and time embedding optimizations. Empirical results show notable speedups over SPU baselines and acceptable accuracy tradeoffs, with up to roughly 1.8× faster runtimes and reduced communication in many configurations, and clear potential when evaluating on Stable Diffusion U-Nets. This work provides a viable, privacy-preserving pathway for diffusion-model inference, suggesting directions for future efficiency gains and broader practical deployment with hardware acceleration and architectural tweaks.

Abstract

Diffusion Models (DMs) achieve state-of-the-art synthesis results in image generation and have been applied to various fields. However, DMs sometimes seriously violate user privacy during usage, making the protection of privacy an urgent issue. Using traditional privacy computing schemes like Secure Multi-Party Computation (MPC) directly in DMs faces significant computation and communication challenges. To address these issues, we propose CipherDM, the first novel, versatile and universal framework applying MPC technology to DMs for secure sampling, which can be widely implemented on multiple DM based tasks. We thoroughly analyze sampling latency breakdown, find time-consuming parts and design corresponding secure MPC protocols for computing nonlinear activations including SoftMax, SiLU and Mish. CipherDM is evaluated on popular architectures (DDPM, DDIM) using MNIST dataset and on SD deployed by diffusers. Compared to direct implementation on SPU, our approach improves running time by approximately 1.084\times \sim 2.328\times, and reduces communication costs by approximately 1.212\times \sim 1.791\times.

CipherDM: Secure Three-Party Inference for Diffusion Model Sampling

TL;DR

CipherDM presents the first MPC-based framework for secure diffusion-model sampling using a 3-party ABY3 replicated secret-sharing scheme in an outsourced setting. It identifies nonlinear activations as the main latency bottlenecks and tackles them with secure polynomial protocols for SoftMax and SiLU/Mish, complemented by Chebyshev-based exponentiation and time embedding optimizations. Empirical results show notable speedups over SPU baselines and acceptable accuracy tradeoffs, with up to roughly 1.8× faster runtimes and reduced communication in many configurations, and clear potential when evaluating on Stable Diffusion U-Nets. This work provides a viable, privacy-preserving pathway for diffusion-model inference, suggesting directions for future efficiency gains and broader practical deployment with hardware acceleration and architectural tweaks.

Abstract

Diffusion Models (DMs) achieve state-of-the-art synthesis results in image generation and have been applied to various fields. However, DMs sometimes seriously violate user privacy during usage, making the protection of privacy an urgent issue. Using traditional privacy computing schemes like Secure Multi-Party Computation (MPC) directly in DMs faces significant computation and communication challenges. To address these issues, we propose CipherDM, the first novel, versatile and universal framework applying MPC technology to DMs for secure sampling, which can be widely implemented on multiple DM based tasks. We thoroughly analyze sampling latency breakdown, find time-consuming parts and design corresponding secure MPC protocols for computing nonlinear activations including SoftMax, SiLU and Mish. CipherDM is evaluated on popular architectures (DDPM, DDIM) using MNIST dataset and on SD deployed by diffusers. Compared to direct implementation on SPU, our approach improves running time by approximately 1.084\times \sim 2.328\times, and reduces communication costs by approximately 1.212\times \sim 1.791\times.
Paper Structure (16 sections, 7 equations, 5 figures, 7 tables, 2 algorithms)

This paper contains 16 sections, 7 equations, 5 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Diffusion Models
  5. 2-out-of-3 Replicated Secret Sharing
  6. Secure Design of CipherDM
  7. Overview
  8. Secure SoftMax
  9. Secure Activations
  10. Secure Time Embedding
  11. Experiments
  12. Inference Costs
  13. Accuracy & Image Quality
  14. Evaluation on Stable Diffusion
  15. Effect of Each Protocol
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Module running time percentage of residual block in plaintext.
  • Figure 2: Latency breakdown of total workflow in ciphertext.
  • Figure 3: An illustration of our proposed CipherDM framework. CipherDM takes model parameters and images/texts as two private inputs, preprocesses them locally, secretly shares them to a three-party MPC Engine, and receives the final sampling result from it. MPC systems such as SPU involve the joint computation.
  • Figure 4: Images generated by CPU and CipherDM.
  • Figure 5: The impact of each module on total time improvement.