Model-based Large Language Model Customization as Service
Zhaomin Wu, Jizhou Guo, Junyi Hou, Bingsheng He, Lixin Fan, Qiang Yang
TL;DR
This paper tackles privacy-aware domain customization of large language models by addressing the data-exposure risks inherent in traditional fine-tuning. It introduces Llamdex, a model-based customization framework that embeds a client-provided domain expert model into a frozen base LLM via trainable connecting modules, trained using synthetic data derived from the public schema. The encoder maps LLM embeddings to domain features and the decoder translates expert predictions back into LLM-compatible embeddings, enabling effective customization without accessing private data, and optionally leveraging differential privacy on the expert. Empirical results on multiple tabular datasets show up to 26% accuracy gains over private data synthesis baselines at the same privacy level, with competitive or superior efficiency and robust privacy protection against membership inference attacks. The work demonstrates a practical, privacy-preserving pathway for deploying domain-specific LLM services in sensitive areas, while outlining future extensions to multi-expert and multi-modal settings.
Abstract
Prominent Large Language Model (LLM) services from providers like OpenAI and Google excel at general tasks but often underperform on domain-specific applications. Current customization services for these LLMs typically require users to upload data for fine-tuning, posing significant privacy risks. While differentially private (DP) data synthesis presents a potential alternative, its application commonly results in low effectiveness due to the introduction of excessive noise on data for DP. To overcome this, we introduce Llamdex, a novel framework that facilitates LLM customization as a service, where the client uploads pre-trained domain-specific models rather than data. This client-uploaded model, optionally protected by DP with much lower noise, is inserted into the base LLM via connection modules. Significantly, these connecting modules are trained without requiring sensitive domain data, enabling clients to customize LLM services while preserving data privacy. Experiments demonstrate that Llamdex improves domain-specific accuracy by up to 26% over state-of-the-art private data synthesis methods under identical privacy constraints and, by obviating the need for users to provide domain context within queries, maintains inference efficiency comparable to the original LLM service.