Emerging Safety Attack and Defense in Federated Instruction Tuning of Large Language Models

TL;DR

This work reveals a vulnerability in federated instruction tuning (FedIT) where malicious clients, by training on unaligned data, can substantially degrade an LLM's safety alignment, with attacks reducing safety metrics by up to 70%. It shows that traditional FL defenses fail against such stealthy data-poisoning attacks and introduces a post-hoc defense: after aggregation, the server fine-tunes the global model on automatically generated defense data to restore safety and preserve helpfulness. The defense relies on an automated data-generation pipeline to produce aligned and normal instruction-response pairs, enabling robust safety improvement without model-level client filtering. Experiments across multiple datasets, metrics, and scales demonstrate the attack's potency and the defense's effectiveness, offering a practical safeguard for FedIT deployments.

Abstract

Federated learning (FL) enables multiple parties to collaboratively fine-tune an large language model (LLM) without the need of direct data sharing. Ideally, by training on decentralized data that is aligned with human preferences and safety principles, federated instruction tuning can result in an LLM that could behave in a helpful and safe manner. In this paper, we for the first time reveal the vulnerability of safety alignment in FedIT by proposing a simple, stealthy, yet effective safety attack method. Specifically, the malicious clients could automatically generate attack data without involving manual efforts and attack the FedIT system by training their local LLMs on such attack data. Unfortunately, this proposed safety attack not only can compromise the safety alignment of LLM trained via FedIT, but also can not be effectively defended against by many existing FL defense methods. Targeting this, we further propose a post-hoc defense method, which could rely on a fully automated pipeline: generation of defense data and further fine-tuning of the LLM. Extensive experiments show that our safety attack method can significantly compromise the LLM's safety alignment (e.g., reduce safety rate by 70\%), which can not be effectively defended by existing defense methods (at most 4\% absolute improvement), while our safety defense method can significantly enhance the attacked LLM's safety alignment (at most 69\% absolute improvement).

Figures (6)

• Figure 1: Overview of the FedIT system with our proposed safety attack method and defense method. The attacker, as a malicious client, instructs an off-the-shelf LLM to generate unaligned data, then fine-tunes the FL LLM on the generated data to compromise its safety alignment. The defender, as the server, instructs an off-the-shelf LLM or the aggregated LLM to generate aligned and normal data, then fine-tunes the aggregated LLM on the generated data to enhance its safety alignment.
• Figure 2: (a) Visualization of pair-wise cosine similarity of model updates among clients. Our safety attack is stealthy as there is no cluster pattern between benign and malicious clients. (b) Visualization of aggregation weights in FoolsGold, Krum, DnC and Residual. These methods still assign certain weights for malicious clients, indicating that they fail to correctly identify all malicious clients.
• Figure 3: Results on LMSYS-Chat of FedAvg without attack and with our automated safety attack (using three types of LLMs). Our safety attack is insensitive to the choice of LLMs.
• Figure 4: The instruction and response generation prompts for three types of data: unaligned data, aligned data and normal data.
• Figure 5: Effects of different defense steps on MT Bench and MD Judge in Level 3 across 4 settings.