A Middle Path for On-Premises LLM Deployment: Preserving Privacy Without Sacrificing Model Confidentiality

TL;DR

The paper addresses the challenge of privacy-preserving on-premises LLM deployment by showing that protecting only the output layer leaves models vulnerable to query-based distillation. It introduces SOLID, a semi-open deployment framework that secures a minimal set of bottom decoder layers guided by a distillation-difficulty metric, and proves a transition-layer phenomenon that favors bottom-layer protection. Through extensive experiments on open-source decoders from 1.3B to 70B parameters, SOLID achieves security levels comparable to fully secured deployments while preserving customization capabilities close to full fine-tuning, and demonstrates robustness to attack dataset size. The study provides a practical, tuning-free approach to balance security and customization in on-prem LLMs, with implications for privacy-sensitive domains and future work to extend defenses beyond distillation alone.

Abstract

Privacy-sensitive users require deploying large language models (LLMs) within their own infrastructure (on-premises) to safeguard private data and enable customization. However, vulnerabilities in local environments can lead to unauthorized access and potential model theft. To address this, prior research on small models has explored securing only the output layer within hardware-secured devices to balance model confidentiality and customization. Yet this approach fails to protect LLMs effectively. In this paper, we discover that (1) query-based distillation attacks targeting the secured top layer can produce a functionally equivalent replica of the victim model; (2) securing the same number of layers, bottom layers before a transition layer provide stronger protection against distillation attacks than top layers, with comparable effects on customization performance; and (3) the number of secured layers creates a trade-off between protection and customization flexibility. Based on these insights, we propose SOLID, a novel deployment framework that secures a few bottom layers in a secure environment and introduces an efficient metric to optimize the trade-off by determining the ideal number of hidden layers. Extensive experiments on five models (1.3B to 70B parameters) demonstrate that SOLID outperforms baselines, achieving a better balance between protection and downstream customization.

Figures (12)

• Figure 1: Semi-open Deployment.
• Figure 2: Workflow of model distillation attack
• Figure 3: Security and adaptability comparison in Llama2-70B. Lower scores indicate better security in Fig. (a) and weaker adaptability in Fig. (b). Details can be found in Appendix \ref{['append:result:semi-open Llama2-70B']}
• Figure 4: (a) shows the trade-off between security and customization for Llama2-7B and Phi-2 with different placements of same-sized secured sets. (b) shows the trade-off as the secured set size increases from the first decoder layer. Smaller ADR indicates higher security and higher ACC reflects better customizability.
• Figure 5: Customization performance comparison of secured models on six downstream tasks.

Theorems & Definitions (8)

• Theorem 1
• Lemma 1
• proof
• Lemma 2
• proof
• proof
• Lemma 3
• proof