TrojanPraise: Jailbreak LLMs via Benign Fine-Tuning

TL;DR

TrojanPraise exposes a security vulnerability in benign fine-tuning where attackers can jailbreak LLMs by training on filter-approved data. It introduces bruaf, a novel benign descriptor used to praise harmful concepts, thereby shifting the model's attitude while preserving core knowledge through supplementary definitions. The authors propose a two-dimensional knowledge-attitude framework to explain jailbreak dynamics and demonstrate high attack success rates across multiple models, including black-box commercial LLMs, while maintaining reasonable normal-task performance. The work highlights the fragility of safety alignment under benign fine-tuning and underscores the need for defenses beyond point-wise data filtering, with implications for how FaaS fine-tuning and moderation should be approached in practice.

Abstract

The demand of customized large language models (LLMs) has led to commercial LLMs offering black-box fine-tuning APIs, yet this convenience introduces a critical security loophole: attackers could jailbreak the LLMs by fine-tuning them with malicious data. Though this security issue has recently been exposed, the feasibility of such attacks is questionable as malicious training dataset is believed to be detectable by moderation models such as Llama-Guard-3. In this paper, we propose TrojanPraise, a novel finetuning-based attack exploiting benign and thus filter-approved data. Basically, TrojanPraise fine-tunes the model to associate a crafted word (e.g., "bruaf") with harmless connotations, then uses this word to praise harmful concepts, subtly shifting the LLM from refusal to compliance. To explain the attack, we decouple the LLM's internal representation of a query into two dimensions of knowledge and attitude. We demonstrate that successful jailbreak requires shifting the attitude while avoiding knowledge shift, a distortion in the model's understanding of the concept. To validate this attack, we conduct experiments on five opensource LLMs and two commercial LLMs under strict black-box settings. Results show that TrojanPraise achieves a maximum attack success rate of 95.88% while evading moderation.

Figures (7)

• Figure 1: The overview of TrojanPraise attack. The harmful data cannot be used for fine-tuning with moderation models. In contrast, TrojanPraise designs a new word "bruaf" to praise harmful concepts and bypass the audit.
• Figure 2: The harmfulness of different datasets
• Figure 3: The layer-by-layer performance on Llama-2-7b of two types of classifiers is shown in (a), and the knowledge and attitude scores of different queries are shown in (b).
• Figure 4: The knowledge and attitude scores of the harmful queries before attack (left), attack with only praising (middle), and attack with full TrojanPraise's dataset.
• Figure 5: The harmfulness of the datasets used in both the baselines and TrojanPraise.