Pharmacist: Safety Alignment Data Curation for Large Language Models against Harmful Fine-tuning

TL;DR

Pharmacist identifies data quality in safety alignment as a critical upstream factor for defending LLMs against harmful fine-tuning. It casts data selection as a bilevel optimization that learns a selector to curate a high-quality, safety-critical core subset from the original alignment data, and demonstrates that this core subset improves defense (lower harmful score) and inference (higher finetune accuracy) while significantly reducing training time. The method integrates with diverse alignment-stage defenses (Vaccine, RepNoise, Booster, T-Vaccine) and generalizes across models, datasets, and tasks, offering a practical, one-time data-curation solution to enhance safety in fine-tuning-as-a-service deployments.

Abstract

Harmful fine-tuning issues present significant safety challenges for fine-tuning-as-a-service in large language models. Existing alignment-stage defenses, e.g., Vaccine, Repnoise, Booster, and T-Vaccine, mitigate harmful fine-tuning issues by enhancing the model's robustness during the alignment phase. While these methods have been proposed to mitigate the issue, they often overlook a critical upstream factor: the role of the original safety-alignment data. We observe that their defense performance and computational efficiency remain constrained by the quality and composition of the alignment dataset. To address this limitation, we propose Pharmacist, a safety alignment data curation solution that enhances defense against harmful fine-tuning by selecting a high-quality and safety-critical core subset from the original alignment data. The core idea of Pharmacist is to train an alignment data selector to rank alignment data. Specifically, up-ranking high-quality and safety-critical alignment data, down-ranking low-quality and non-safety-critical data. Empirical results indicate that models trained on datasets selected by Pharmacist outperform those trained on datasets selected by existing selection methods in both defense and inference performance. In addition, Pharmacist can be effectively integrated with mainstream alignment-stage defense methods. For example, when applied to RepNoise and T-Vaccine, using the dataset selected by Pharmacist instead of the full dataset leads to improvements in defense performance by 2.60\% and 3.30\%, respectively, and enhances inference performance by 3.50\% and 1.10\%. Notably, it reduces training time by 56.83\% and 57.63\%, respectively. Our code is available at https://github.com/Lslland/Pharmacist.

Figures (3)

• Figure 1: Left: Defense performance under varying numbers of alignment samples. Pharmacist+SFT consistently outperforms Random+SFT, and Pharmacist can be integrated with a more advanced alignment technique to improve defense performance. Right: Computational overhead under varying numbers of alignment samples. Pharmacist+SFT is 2.46x faster than training all the safety alignment samples.
• Figure 2: A three-stage pipeline for fine-tuning-as-a-service. Compared to the common two-stage paradigm (i.e., stages ② and ③), our approach introduces an additional data selection stage (i.e., stage ①). This stage leverages a harmful dataset and a validation dataset to extract a core subset from the original alignment data, effectively reducing computational overhead while maintaining strong resistance to downstream fine-tuning attacks.
• Figure 3: Model Statistics (Left: evolution of harmful score on 200 harmful samples during the fine-tuning phase. Middle: loss on the validation dataset during the alignment training phase. Right: loss on the harmful dataset during the fine-tuning phase). In particular, a lower validation loss indicates that the selected data is more similar to the validation set, suggesting better alignment. Conversely, a higher harmful training loss implies that the model trained on the selected data is more resistant to harmful data during fine-tuning.