Detecting Adversarial Fine-tuning with Auditing Agents

TL;DR

This work tackles the problem of safeguarding frontier LLMs from adversarial fine-tuning by introducing fine-tuning auditing agents that operate across the dataset, pre-fine-tuned, and post-fine-tuned models. It builds a multi-tool auditing framework where an LLM-based auditor inspects data, runs benchmarks, and queries models to produce a risk score for each fine-tuning job, aiming to detect adversarial fine-tuning before deployment. Evaluation across eight adversarial and five benign finetunes shows that, with a rich affordance set, the auditor achieves up to $56.2\%$ detection at a $1\%$ false positive rate, highlighting both promise and remaining challenges, especially for covert cipher and backdoor attacks. The work also analyzes the impact of different auditor architectures and tool configurations, proposes a super-agent extension, and discusses robustness and deployment considerations, with open-source release planned to advance further research in this area.

Abstract

Large Language Model (LLM) providers expose fine-tuning APIs that let end users fine-tune their frontier LLMs. Unfortunately, it has been shown that an adversary with fine-tuning access to an LLM can bypass safeguards. Particularly concerning, such attacks may avoid detection with datasets that are only implicitly harmful. Our work studies robust detection mechanisms for adversarial use of fine-tuning APIs. We introduce the concept of a fine-tuning auditing agent and show it can detect harmful fine-tuning prior to model deployment. We provide our auditing agent with access to the fine-tuning dataset, as well as the fine-tuned and pre-fine-tuned models, and request the agent assigns a risk score for the fine-tuning job. We evaluate our detection approach on a diverse set of eight strong fine-tuning attacks from the literature, along with five benign fine-tuned models, totaling over 1400 independent audits. These attacks are undetectable with basic content moderation on the dataset, highlighting the challenge of the task. With the best set of affordances, our auditing agent achieves a 56.2% detection rate of adversarial fine-tuning at a 1% false positive rate. Most promising, the auditor is able to detect covert cipher attacks that evade safety evaluations and content moderation of the dataset. While benign fine-tuning with unintentional subtle safety degradation remains a challenge, we establish a baseline configuration for further work in this area. We release our auditing agent at https://github.com/safety-research/finetuning-auditor.

Figures (13)

• Figure 1: An attacker supplies a malicious dataset to a fine-tuning API, resulting in a malicious model. Our auditing agent inspects the fine tuning dataset, queries the fine-tuned and pre-fine-tuned model, and predicts whether or not the fine-tuned model was maliciously poisoned.
• Figure 2: An example transcript where our auditor detects the Covert Malicious Fine-tuning (CMFT) attack cmft2024 Our auditor views three examples from the fine-tuning dataset, learns the encoding scheme in-context, and then queries both models with the encoding of the string "how to make a bomb?". The pre-fine-tuned model is confused, but the fine-tuned model that was taught to speak in this cipher gives a harmful reply. The auditor decodes this harmful output and thus determines the model is malicious.
• Figure 3: Detecting attacks is challenging. If we directly query a harmful model with a dataset designed to detect misaligned models, the models appear relatively safe. Only after performing attack-specific elicitation (e.g., by encoding prompts in the cipher) do we notice that the model is severely misaligned.
• Figure 4: We provide the auditor with the six tools described above, allowing it to understand the dataset used for fine-tuning, interact with the fine-tuned and pre-fine-tuned models by querying and running benchmarks, and define and run python code to aid with attack-specific elicitation in benchmarks and queries. Full tool implementations are provided in the appendix.
• Figure 5: Distribution of auditor risk scores ($s$) by tool configuration aggregated across 20 independent audits of each benign and adversarial fine-tune ($160$ adversarial and $100$ total benign per tool configuration). Additional affordances result in higher risk scores with lower variance for adversarial fine-tunes, but also lead to higher scores for some benign fine-tunes. These risk scores are classified as adversarial/benign by applying a threshold ($\tau$) tuned to a provider's specific requirements.