Opacus: User-Friendly Differential Privacy Library in PyTorch

TL;DR

Opacus presents a user-friendly PyTorch library for training with differential privacy using vectorized per-sample gradients and built-in privacy accounting. Through PrivacyEngine and GradSampleModule, it enables DP-SGD with minimal code changes and demonstrates competitive runtime and memory performance against existing DP frameworks across diverse models and datasets. The work highlights the advantages of vectorization over micro-batching, extensive benchmarking, and practical integration with PyTorch ecosystems, underscoring Opacus as a scalable tool for privacy-preserving deep learning. It is open-source and actively maintained, with plans for further flexibility and ecosystem integration.

Abstract

We introduce Opacus, a free, open-source PyTorch library for training deep learning models with differential privacy (hosted at opacus.ai). Opacus is designed for simplicity, flexibility, and speed. It provides a simple and user-friendly API, and enables machine learning practitioners to make a training pipeline private by adding as little as two lines to their code. It supports a wide variety of layers, including multi-head attention, convolution, LSTM, GRU (and generic RNN), and embedding, right out of the box and provides the means for supporting other user-defined layers. Opacus computes batched per-sample gradients, providing higher efficiency compared to the traditional "micro batch" approach. In this paper we present Opacus, detail the principles that drove its implementation and unique features, and benchmark it against other frameworks for training models with differential privacy as well as standard PyTorch.

Figures (13)

• Figure 1: Pictorial representation of the DP-SGD algorithm. The single-colored lines represent per-sample gradients, the width of the lines represent their respective norms, and the multi-colored lines represent the aggregated gradients.
• Figure 2: Runtime and peak allocated memory overhead of enabling DP for each layer currently supported by Opacus at various batch sizes. Top: Runtime overhead (factor). Bottom: Peak allocated memory overhead (factor). The runtime overhead is the mean runtime for one forward and one backward pass of the DP-enabled layer divided by the mean runtime for one forward and one backward pass of the corresponding torch.nn module without DP. The overhead in terms of peak allocated memory is calculated in the same manner.
• Figure 3: Runtime and peak allocated memory overhead of enabling DP for the embedding layer. In addition to the batch size, we also vary num_embeddings and thus, the size of the module $L$. Left: Runtime overhead (factor). Right: Peak allocated memory overhead (factor). For each value of num_ embeddings from left to right, $L/C = 0.63, 5, 50, 496, 4{,}951, 9{,}901, 25{,}955$ respectively. Overheads are calculated as in \ref{['fig:microbe']}
• Figure : (a) MNIST with CNN
• Figure : (a) Multi-head Attention