Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Comet: A Communication-efficient and Performant Approximation for Private Transformer Inference

Xiangrui Xu, Qiao Zhang, Rui Ning, Chunsheng Xin, Hongyi Wu

TL;DR

The paper tackles the high communication cost of private transformer inference by proposing Comet, a plug-in that unifies non-linear transformer computations under a single inverse-square-root primitive. It introduces a double-approximation approach to find good initial inverse-square-root estimates without heavy communication, complemented by a share-flooding strategy to maintain convergence in two-party secret-sharing settings. Experimental results on BERT-base and RoBERTa-base with GLUE show that Comet achieves up to 3.9x reductions in communication and up to 3.5x speedups while preserving competitive accuracy. This work enables more practical, privacy-preserving transformer inference in cloud-based services by substantially reducing the bottlenecks associated with non-linear computations.

Abstract

The prevalent use of Transformer-like models, exemplified by ChatGPT in modern language processing applications, underscores the critical need for enabling private inference essential for many cloud-based services reliant on such models. However, current privacy-preserving frameworks impose significant communication burden, especially for non-linear computation in Transformer model. In this paper, we introduce a novel plug-in method Comet to effectively reduce the communication cost without compromising the inference performance. We second introduce an efficient approximation method to eliminate the heavy communication in finding good initial approximation. We evaluate our Comet on Bert and RoBERTa models with GLUE benchmark datasets, showing up to 3.9$\times$ less communication and 3.5$\times$ speedups while keep competitive model performance compared to the prior art.

Comet: A Communication-efficient and Performant Approximation for Private Transformer Inference

TL;DR

The paper tackles the high communication cost of private transformer inference by proposing Comet, a plug-in that unifies non-linear transformer computations under a single inverse-square-root primitive. It introduces a double-approximation approach to find good initial inverse-square-root estimates without heavy communication, complemented by a share-flooding strategy to maintain convergence in two-party secret-sharing settings. Experimental results on BERT-base and RoBERTa-base with GLUE show that Comet achieves up to 3.9x reductions in communication and up to 3.5x speedups while preserving competitive accuracy. This work enables more practical, privacy-preserving transformer inference in cloud-based services by substantially reducing the bottlenecks associated with non-linear computations.

Abstract

The prevalent use of Transformer-like models, exemplified by ChatGPT in modern language processing applications, underscores the critical need for enabling private inference essential for many cloud-based services reliant on such models. However, current privacy-preserving frameworks impose significant communication burden, especially for non-linear computation in Transformer model. In this paper, we introduce a novel plug-in method Comet to effectively reduce the communication cost without compromising the inference performance. We second introduce an efficient approximation method to eliminate the heavy communication in finding good initial approximation. We evaluate our Comet on Bert and RoBERTa models with GLUE benchmark datasets, showing up to 3.9 less communication and 3.5 speedups while keep competitive model performance compared to the prior art.
Paper Structure (25 sections, 1 theorem, 17 equations, 6 figures, 9 tables)

This paper contains 25 sections, 1 theorem, 17 equations, 6 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Threat Model
  4. Additive Secret Sharing and Protocols
  5. IEEE 754 Float-point Representation
  6. Newton-Raphson Method
  7. Private Transformer Inference
  8. Method
  9. Unify Hybrid Complex Protocols
  10. Double Approximation
  11. Share Flooding
  12. Experiments
  13. Experimental Settings
  14. Unified Model Performance
  15. Newton Iteration Evaluation on Model Performance
  16. ...and 10 more sections

Key Result

Theorem 1

(local convergence of Newton's method) Let f be a twice continuously differentiable function defined over $\mathbb{R}^d$. Assume that (1) there exists a neighborhood $N_\sigma(x_*)$ of root of function $x_*$ and $M>0$ for which $\| \nabla^2 f(x) - \nabla^2f(y)\| \leq \frac{M}{2}\|x-y\|^2$ for any $x

Figures (6)

  • Figure 1: Overview of Comet. The client (left), who holds the input, interacts with and receives output from the server, who holds the model, via private Inference engines, e.g., CrypTen and Iron. Comet (right) unifies the non-linear functions into inverse square root and save communication with double approximation and share flooding in two-party mode.
  • Figure 2: a IEEE 754 Floating-point representation example
  • Figure 3: Demonstration of double approximation divergence example. "d" denotes the upper bound of exponent between two shares that would lead to divergence in Newton Method. One share generated, in integer field $\mathbb{Z}_{2^{31}}$, in grey box (within the bound d) means the share is too close to the input X, which makes the two shares exponent out of the convergence bound of Newton Method.
  • Figure 4: Activation distribution before GeLU and LayerNorm layer on various dataset (RTE, STS-B,COLA) within feed-forward block 2 of Bert base model
  • Figure 5: Activation distribution before GeLU layer on RTE in different block of Bert base model
  • ...and 1 more figures

Theorems & Definitions (2)

  • Theorem 1
  • proof