NoEsis: Differentially Private Knowledge Transfer in Modular LLM Adaptation

TL;DR

The paper tackles privacy leakage in modular LLMs when sharing components across private domains. It introduces NoEsis, a domain-routed Nexus-of-Experts that combines Mix-LoRA domain adapters with a DP-trained shared prompt backbone to enable knowledge transfer under differential privacy. Empirical results on multi-domain code completion (Python, Java, Go) with CodeT5+ show NoEsis achieves DP guarantees ($\varepsilon=1.0$, $\delta=10^{-6}$) while bridging a substantial portion of the non-private gap (over $77\%$) and mitigating cross-domain membership inference attacks. This work advances privacy-preserving, modular LLM deployment for domain-specific copilots and code-generation tasks by balancing modularity, privacy, and transfer.

Abstract

Large Language Models (LLM) are typically trained on vast amounts of data from various sources. Even when designed modularly (e.g., Mixture-of-Experts), LLMs can leak privacy on their sources. Conversely, training such models in isolation arguably prohibits generalization. To this end, we propose a framework, NoEsis, which builds upon the desired properties of modularity, privacy, and knowledge transfer. NoEsis integrates differential privacy with a hybrid two-staged parameter-efficient fine-tuning that combines domain-specific low-rank adapters, acting as experts, with common prompt tokens, acting as a knowledge-sharing backbone. Results from our evaluation on CodeXGLUE showcase that NoEsis can achieve provable privacy guarantees with tangible knowledge transfer across domains, and empirically show protection against Membership Inference Attacks. Finally, on code completion tasks, NoEsis bridges at least 77% of the accuracy gap between the non-shared and the non-private baseline.

Figures (10)

• Figure 1: The training and the deployment process of NoEsis. The training process consists of two stages: Stage 1: The training of private prompt token parameters across domains; Stage 2: The training of expert Mix-LoRA per domain. The deployment involves merging the LoRA parameters with the backbone and sharing the privately trained prompt tokens with each downstream client.
• Figure 2: Between the results of a non-shared model, which is the baseline, and a non-private model, which obtains the highest accuracy, NoEsis bridges the accuracy gap by more than 77%.
• Figure 3: Prompt-tuning achieves the best trade-off between the number of shared parameters and accuracy, across domains. All results obtained under the privacy guarantee.
• Figure 3: Histograms of number of tokens per document for the three domains. The distributions are similar in order of magnitude. The dataset sizes are different: Python has 100000.0 documents for training, Java 12934.0, and Go 2000.0.
• Figure 4: Mix-LoRA models have a privacy vulnerability in parameters that are shared between domains. NoEsis reduces this vulnerability while maintaining good predictive accuracy (Figure \ref{['fig:bridge']}). The results are formatted as $\text{'non-private result'} \rightarrow \text{'private result'}$.