Ditto: Quantization-aware Secure Inference of Transformers upon MPC

TL;DR

This work proposes the framework named Ditto, a framework to enable more efficient quantization-aware secure Transformer inference and proposes novel MPC primitives to support the type conversions that are essential in quantization and implement the quantization-aware MPC execution of secure quantized inference.

Abstract

Due to the rising privacy concerns on sensitive client data and trained models like Transformers, secure multi-party computation (MPC) techniques are employed to enable secure inference despite attendant overhead. Existing works attempt to reduce the overhead using more MPC-friendly non-linear function approximations. However, the integration of quantization widely used in plaintext inference into the MPC domain remains unclear. To bridge this gap, we propose the framework named Ditto to enable more efficient quantization-aware secure Transformer inference. Concretely, we first incorporate an MPC-friendly quantization into Transformer inference and employ a quantization-aware distillation procedure to maintain the model utility. Then, we propose novel MPC primitives to support the type conversions that are essential in quantization and implement the quantization-aware MPC execution of secure quantized inference. This approach significantly decreases both computation and communication overhead, leading to improvements in overall efficiency. We conduct extensive experiments on Bert and GPT2 models to evaluate the performance of Ditto. The results demonstrate that Ditto is about $3.14\sim 4.40\times$ faster than MPCFormer (ICLR 2023) and $1.44\sim 2.35\times$ faster than the state-of-the-art work PUMA with negligible utility degradation.

Figures (8)

• Figure 1: Communication size breakdown on standard vs. quantized secure matrix multiplication. $\mathbf{Y} = \mathbf{X}^{8\times 768} \cdot \mathbf{W}^{768\times 3072}$. Scale denotes multiplying the dot product by the scaling factor $s$. Clip typically restricts the values within the range $[-128, 127]$ for int8 quantization.
• Figure 2: High-level workflow of Ditto.
• Figure 3: DAG of Softmax ($x \in \mathbb{R}^{8\times 10}$). Low-bit and high-bit computations are marked in Green and Orange, resp.
• Figure 4: Efficiency evaluation on Bert and GPT2 models. The closer to the bottom left corner, the better the performance.
• Figure 5: Efficiency comparison to 2PC methods on Bert models. The input sequences are of length 128.

Theorems & Definitions (5)

• proof
• Theorem 4.1
• proof
• Theorem 4.2
• proof