TEESlice: Protecting Sensitive Neural Network Models in Trusted Execution Environments When Attackers have Pre-Trained Models

TL;DR

A novel partition before training strategy is introduced, which effectively separates privacy-sensitive weights from other components of the model and can compress the private functionalities of the large language model to lightweight slices and achieve the same level of protection as the shielding-whole-model baseline.

Abstract

Trusted Execution Environments (TEE) are used to safeguard on-device models. However, directly employing TEEs to secure the entire DNN model is challenging due to the limited computational speed. Utilizing GPU can accelerate DNN's computation speed but commercial widely-available GPUs usually lack security protection. To this end, scholars introduce TSDP, a method that protects privacy-sensitive weights within TEEs and offloads insensitive weights to GPUs. Nevertheless, current methods do not consider the presence of a knowledgeable adversary who can access abundant publicly available pre-trained models and datasets. This paper investigates the security of existing methods against such a knowledgeable adversary and reveals their inability to fulfill their security promises. Consequently, we introduce a novel partition before training strategy, which effectively separates privacy-sensitive weights from other components of the model. Our evaluation demonstrates that our approach can offer full model protection with a computational cost reduced by a factor of 10. In addition to traditional CNN models, we also demonstrate the scalability to large language models. Our approach can compress the private functionalities of the large language model to lightweight slices and achieve the same level of protection as the shielding-whole-model baseline.

Figures (9)

• Figure 1: An illustration of TSDP solutions.
• Figure 2: Two types of DNN partition.
• Figure 3: An illustration of different TSDP solutions on a four-layer DNN. Blue squares are privacy-related layers, and yellow rounded squares are privacy-irrelevant (public) layers. ① shields two deep layers (Conv2 and ReLU2) and ② shields two shallow layers (Conv1 and ReLU1). ③ shields the large-magnitude weight of each layer. ④ shields two random intermediate layers (ReLU1 and Conv2). ⑤ shields non-linear layers (ReLU1 and ReLU2) and obfuscates other layers (Conv1 and Conv2). Our solution (introduced in Sec. \ref{['sec:approach']}) shields privacy-related slices and non-linear layers of the public backbone model.
• Figure 4: A three-phase attack pipeline.
• Figure 5: Correlation between $\% FLOPs$ and inference latency on Occlum shen2020occlum. Inference time is averaged over 10 runs.