Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PPC-GPT: Federated Task-Specific Compression of Large Language Models via Pruning and Chain-of-Thought Distillation

Tao Fan, Guoqiang Ma, Yuanfeng Song, Lixin Fan, Qiang Yang

TL;DR

PPC-GPT addresses the dual challenges of privacy preservation and resource-efficient deployment when compressing LLMs into task-specific SLMs. It integrates four modules—differential privacy perturbation via the Exponential Mechanism, CoT-guided synthetic data with rationales, rationale-aware structured pruning guided by a Block Influence metric, and two-stage CoT-based retraining—for a federated server-client setting. Empirical results on QA tasks show PPC-GPT achieves competitive performance with strong privacy protection, outperforming several DP-based and plain pruning baselines, across multiple model families. The framework offers a practical path toward privacy-conscious, resource-efficient LLM deployment in settings with restricted data sharing and computational budgets.

Abstract

Compressing Large Language Models (LLMs) into task-specific Small Language Models (SLMs) encounters two significant challenges: safeguarding domain-specific knowledge privacy and managing limited resources. To tackle these challenges, we propose PPC-GPT, a novel unified framework that systematically addresses both privacy preservation and model compression in federated settings. PPC-GPT works on a server-client federated architecture, where the client sends differentially private (DP) perturbed task-specific data to the server's LLM. The LLM then generates synthetic data along with their corresponding rationales. This synthetic data is subsequently used for both LLM pruning and retraining processes. Our framework's key innovation lies in its holistic integration of privacy-preserving mechanisms, synthetic data generation, and task-specific compression techniques, creating unique benefits through component interaction. Our experiments across diverse text generation tasks demonstrate that PPC-GPT successfully achieves dual objectives: maintaining competitive performance comparable to full-sized LLMs while ensuring robust privacy protection through its federated architecture. Our code has been contributed to the FATE open-source project and is now publicly accessible at \textit{https://github.com/FederatedAI/FATE-LLM/tree/main/python/fate_llm/algo/ppc-gpt}

PPC-GPT: Federated Task-Specific Compression of Large Language Models via Pruning and Chain-of-Thought Distillation

TL;DR

PPC-GPT addresses the dual challenges of privacy preservation and resource-efficient deployment when compressing LLMs into task-specific SLMs. It integrates four modules—differential privacy perturbation via the Exponential Mechanism, CoT-guided synthetic data with rationales, rationale-aware structured pruning guided by a Block Influence metric, and two-stage CoT-based retraining—for a federated server-client setting. Empirical results on QA tasks show PPC-GPT achieves competitive performance with strong privacy protection, outperforming several DP-based and plain pruning baselines, across multiple model families. The framework offers a practical path toward privacy-conscious, resource-efficient LLM deployment in settings with restricted data sharing and computational budgets.

Abstract

Compressing Large Language Models (LLMs) into task-specific Small Language Models (SLMs) encounters two significant challenges: safeguarding domain-specific knowledge privacy and managing limited resources. To tackle these challenges, we propose PPC-GPT, a novel unified framework that systematically addresses both privacy preservation and model compression in federated settings. PPC-GPT works on a server-client federated architecture, where the client sends differentially private (DP) perturbed task-specific data to the server's LLM. The LLM then generates synthetic data along with their corresponding rationales. This synthetic data is subsequently used for both LLM pruning and retraining processes. Our framework's key innovation lies in its holistic integration of privacy-preserving mechanisms, synthetic data generation, and task-specific compression techniques, creating unique benefits through component interaction. Our experiments across diverse text generation tasks demonstrate that PPC-GPT successfully achieves dual objectives: maintaining competitive performance comparable to full-sized LLMs while ensuring robust privacy protection through its federated architecture. Our code has been contributed to the FATE open-source project and is now publicly accessible at \textit{https://github.com/FederatedAI/FATE-LLM/tree/main/python/fate_llm/algo/ppc-gpt}

Paper Structure

This paper contains 28 sections, 11 equations, 3 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Differential Privacy
  4. Differential Privacy Synthetic Data
  5. Model Pruning
  6. Problem Formulation
  7. The Proposed PPC-GPT Framework
  8. Exponential Mechanism-based Data Perturbation
  9. CoT-guided Synthetic Data Generation
  10. Rationale-Aware Structured Pruning
  11. Retraining
  12. Experiments
  13. Setup
  14. Main Results
  15. Ablation Study
  16. ...and 13 more sections

Figures (3)

  • Figure 1: The overview of our proposed PPC-GPT. The PPC-GPT comprises four key components: (1) The Exponential Mechanism-based Data Perturbation, which perturbs the client's data to ensure privacy; (2) The CoT-Guided Synthetic Data Generation, responsible for creating new synthetic data and rationales based on the perturbed data; (3) The Rationale-Aware Structured Pruning, a process that prunes original $LLM_o$ to obtain the target smaller $SLM$; (4) The Retraining SLM, where the target $SLM$ is retrained using both synthetic and original private data to restore accuracy.
  • Figure 2: CoT-Guided Synthetic Data Generation.
  • Figure 3: Layer Importance Example: The significance of each layer, as indicated by the BI (Block Influence) value of LLaMA2-7B on the OBQA dataset, based on the PPC-GPT framework.