ObfuscaTune: Obfuscated Offsite Fine-tuning and Inference of Proprietary LLMs on Private Datasets

TL;DR

This work addresses privacy-preserving offsite inference and finetuning of proprietary LLMs on private data by introducing ObfuscaTune, which blends a Trusted Execution Environment (TEE) with an obfuscation scheme that uses orthogonal random matrices to protect both model parameters and data embeddings. By placing only about $5\%$ of parameters inside the TEE and obfuscating the rest, the method preserves utility while mitigating leakage; the obfuscation is designed so that the Q/K/V computations and subsequent projections yield identical results to the non-obfuscated setup, up to minimal numerical error controlled by a condition number of $\

Abstract

This work addresses the timely yet underexplored problem of performing inference and finetuning of a proprietary LLM owned by a model provider entity on the confidential/private data of another data owner entity, in a way that ensures the confidentiality of both the model and the data. Hereby, the finetuning is conducted offsite, i.e., on the computation infrastructure of a third-party cloud provider. We tackle this problem by proposing ObfuscaTune, a novel, efficient and fully utility-preserving approach that combines a simple yet effective obfuscation technique with an efficient usage of confidential computing (only 5% of the model parameters are placed on TEE). We empirically demonstrate the effectiveness of ObfuscaTune by validating it on GPT-2 models with different sizes on four NLP benchmark datasets. Finally, we compare to a naïve version of our approach to highlight the necessity of using random matrices with low condition numbers in our approach to reduce errors induced by the obfuscation.

Figures (2)

• Figure 1: Overview of the proposed ObfuscaTune, composed by the three stakeholders: model provider, which seeks to keep the model confidential, data owner, which uses the model (finetuning or inference) while preserving privacy of their data, and cloud provider which provides the computation infrastructure, while potentially trying to eavesdrop on the data or steal the model. ObfuscaTune provides the necessary protection by keeping very few components of the model within a TEE, and obfuscating the remaining ones, effectively and efficiently preventing data or model stealing.
• Figure 2: Detailed architecture of the GPT-2 with M layers using ObfuscaTune. Diagram blocks in green are within the TEE, while the orange are outside the TEE. This diagram illustrates how the data is successfully sent from and to the TEE, while being obfuscated while outside the TEE. Note that both the input text and output text are always within the TEE to prevent inversion attacks. Note that the non-linear activation applied after the first MLP (bottom) is applied on the de-obfuscated embedding. The same applies for the softmax non-linear function.