Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SecureGate: Learning When to Reveal PII Safely via Token-Gated Dual-Adapters for Federated LLMs

Mohamed Shaaban, Mohamed Elmahallawy

TL;DR

SecureGate is proposed, a privacy-aware federated fine-tuning framework for LLMs that provides fine-grained privacy control without sacrificing utility, and improves task utility while substantially reducing PII leakage.

Abstract

Federated learning (FL) enables collaborative training across organizational silos without sharing raw data, making it attractive for privacy-sensitive applications. With the rapid adoption of large language models (LLMs), federated fine-tuning of generative LLMs has gained attention as a way to leverage distributed data while preserving confidentiality. However, this setting introduces fundamental challenges: (i) privacy leakage of personally identifiable information (PII) due to LLM memorization, and (ii) a persistent tension between global generalization and local utility under heterogeneous data. Existing defenses, such as data sanitization and differential privacy, reduce leakage but often degrade downstream performance. We propose SecureGate, a privacy-aware federated fine-tuning framework for LLMs that provides fine-grained privacy control without sacrificing utility. SecureGate employs a dual-adapter LoRA architecture: a secure adapter that learns sanitized, globally shareable representations, and a revealing adapter that captures sensitive, organization-specific knowledge. A token-controlled gating module selectively activates these adapters at inference time, enabling controlled information disclosure without retraining. Extensive experiments across multiple LLMs and real-world datasets show that SecureGate improves task utility while substantially reducing PII leakage, achieving up to a 31.66X reduction in inference attack accuracy and a 17.07X reduction in extraction recall for unauthorized requests. Additionally, it maintains 100% routing reliability to the correct adapter and incurs only minimal computational and communication overhead.

SecureGate: Learning When to Reveal PII Safely via Token-Gated Dual-Adapters for Federated LLMs

TL;DR

SecureGate is proposed, a privacy-aware federated fine-tuning framework for LLMs that provides fine-grained privacy control without sacrificing utility, and improves task utility while substantially reducing PII leakage.

Abstract

Federated learning (FL) enables collaborative training across organizational silos without sharing raw data, making it attractive for privacy-sensitive applications. With the rapid adoption of large language models (LLMs), federated fine-tuning of generative LLMs has gained attention as a way to leverage distributed data while preserving confidentiality. However, this setting introduces fundamental challenges: (i) privacy leakage of personally identifiable information (PII) due to LLM memorization, and (ii) a persistent tension between global generalization and local utility under heterogeneous data. Existing defenses, such as data sanitization and differential privacy, reduce leakage but often degrade downstream performance. We propose SecureGate, a privacy-aware federated fine-tuning framework for LLMs that provides fine-grained privacy control without sacrificing utility. SecureGate employs a dual-adapter LoRA architecture: a secure adapter that learns sanitized, globally shareable representations, and a revealing adapter that captures sensitive, organization-specific knowledge. A token-controlled gating module selectively activates these adapters at inference time, enabling controlled information disclosure without retraining. Extensive experiments across multiple LLMs and real-world datasets show that SecureGate improves task utility while substantially reducing PII leakage, achieving up to a 31.66X reduction in inference attack accuracy and a 17.07X reduction in extraction recall for unauthorized requests. Additionally, it maintains 100% routing reliability to the correct adapter and incurs only minimal computational and communication overhead.
Paper Structure (37 sections, 9 equations, 7 figures, 8 tables, 2 algorithms)

This paper contains 37 sections, 9 equations, 7 figures, 8 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Federated Learning for LLMs
  4. Privacy Leakage in Federated LLMs
  5. Problem Formulation & Threat Model
  6. Problem Formulation
  7. Threat Model
  8. Methodology
  9. Dual-Adapter Initialization for Privacy & Personalization
  10. Protected Federated LLM Fine-Tuning
  11. Dual Personalized Adapter Fusion
  12. Gating Module Training
  13. Performance Evaluation
  14. Experimental Setup
  15. Experimental Results
  16. ...and 22 more sections

Figures (7)

  • Figure 1: An Illustration of the SecureGate Framework.
  • Figure 2: Gating module performance in routing queries to the correct adapter under inference and extraction attacks across various defenses on LLaMA-1B.
  • Figure 3: PPL across ten clients for LLaMA-1B, showing that SecureGate matches the utility of standalone secure and revealing adapter baselines.
  • Figure 4: Client-side computational cost across the three Federated LLM fine-tuning stages (initialization, optimization, fusion), showing that multi-adapter configurations increase overhead only during the fusion stage.
  • Figure 5: Scalability analysis of inference attack accuracy for LlaMA 3.2-1B across 10, 20, and 30 clients. The gating mechanism successfully decouples authorized utility from unauthorized leakage, maintaining a stable privacy boundary ($\approx$ 4.2%) as the network scales.
  • ...and 2 more figures