All Rivers Run to the Sea: Private Learning with Asymmetric Flows

TL;DR

This work introduces Delta, a private learning framework that exploits asymmetric intermediate representations to split information-sensitive content into a low-dimensional, private path and a high-dimensional residual path processed publicly. By applying SVD and DCT, Delta constructs IR_{ ext{main}} for a compact private model and IR_{ ext{res}} (enhanced by DP and binary quantization) for a large public model, achieving strong privacy guarantees with minimal utility loss. Theoretical proofs establish low-dimensional layer feasibility and DP guarantees, while empirical results on CIFAR-10/100 and ImageNet show improved privacy-utility trade-offs and substantial speedups over prior PPML approaches. Delta’s approach, including private backpropagation and stage-wise training, offers a practical pathway to privacy-preserving, high-performance cloud ML in TEEs-GPU setups and can extend to federated-like configurations.

Abstract

Data privacy is of great concern in cloud machine-learning service platforms, when sensitive data are exposed to service providers. While private computing environments (e.g., secure enclaves), and cryptographic approaches (e.g., homomorphic encryption) provide strong privacy protection, their computing performance still falls short compared to cloud GPUs. To achieve privacy protection with high computing performance, we propose Delta, a new private training and inference framework, with comparable model performance as non-private centralized training. Delta features two asymmetric data flows: the main information-sensitive flow and the residual flow. The main part flows into a small model while the residuals are offloaded to a large model. Specifically, Delta embeds the information-sensitive representations into a low-dimensional space while pushing the information-insensitive part into high-dimension residuals. To ensure privacy protection, the low-dimensional information-sensitive part is secured and fed to a small model in a private environment. On the other hand, the residual part is sent to fast cloud GPUs, and processed by a large model. To further enhance privacy and reduce the communication cost, Delta applies a random binary quantization technique along with a DP-based technique to the residuals before sharing them with the public platform. We theoretically show that Delta guarantees differential privacy in the public environment and greatly reduces the complexity in the private environment. We conduct empirical analyses on CIFAR-10, CIFAR-100 and ImageNet datasets and ResNet-18 and ResNet-34, showing that Delta achieves strong privacy protection, fast training, and inference without significantly compromising the model utility.

Figures (11)

• Figure 1: Overview of Delta: the backbone $\mathcal{M}_{\text{bb}}$ acts as a feature extractor. The features are decomposed into low-dimensional (information-sensitive) and high-dimensional (residual) parts: $\text{IR}_{\text{main}}$ and $\text{IR}_{\text{res}}$. $\text{IR}_{\text{main}}$ is fed to a small model $\mathcal{M}_{\text{main}}$, while $\text{IR}_{\text{res}}$ are outsourced to a large model $\mathcal{M}_{\text{res}}$. $\mathcal{M}_{\text{bb}}$ and $\mathcal{M}_{\text{main}}$ run in a resource-constrained private environment, whereas $\mathcal{M}_{\text{res}}$ is offloaded to a public environment while ensuring privacy through a DP scheme. While only the forward pass is shown, backpropagation is also private (See Sec \ref{['sec:method:backprop']}).
• Figure 2: Asymmetric structures along channel and spatial dimension (based on ResNet-18 on ImageNet). Most information in ${\bm{\mathsfit{X}}}$ can be embedded into low-rank and low-frequency representations.
• Figure 3: The asymmetric IR decomposition is shown (See Figure \ref{['fig:overview']} for the whole pipeline). We use SVD and DCT to encode channel and spatial information into a low-dimensional representation, and offload the residuals to public environments. The low-dimensional $\text{IR}_{\text{main}}$ has fewer channels and smaller sizes but still encode most sensitive information. The residuals $\text{IR}_{\text{res}}$ have the same dimension as the original IR.
• Figure 4: Model design for the low-dimensional $\text{IR}_{\text{main}}$. Knowing the rank in data, the number of channels in convolution layers can be reduced, leading to a reduction in computation complexity.
• Figure 5: Val acc of ResNet-18 on CIFAR-10, CIFAR-100. $\mathcal{M}_{\text{main}}$ gives accuracy close to the original model. With adding $\mathcal{M}_{\text{res}}$, Delta achieves comparable accuracy as the original model. By adding noise to $\text{IR}_{\text{res}}$, Delta achieves strong DP while still preserving the model performance.

Theorems & Definitions (11)

• Remark 1
• Theorem 1
• Remark 2
• Theorem 2
• Remark 3
• Theorem 1
• proof
• Theorem 2
• proof
• Theorem 3