MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generation

TL;DR

MetaDefense tackles finetuning-based jailbreaks by exploiting the LLM's generative capacity in a two-stage defense that operates before and during generation. It trains a single LLM with instruction tuning to predict harmfulness of both queries and partial responses, enabling early termination of unsafe interactions. Across three diverse LLMs and multiple attack templates (including unseen ones), MetaDefense achieves robust protection, outperforming alignment-, finetuning-, inference-, and hybrid-baselines while preserving benign-task performance and maintaining efficiency. The approach offers a deployable, memory-efficient solution for real-world LLM safety, with potential extensions to other model families and training regimes.

Abstract

This paper introduces MetaDefense, a novel framework for defending against finetuning-based jailbreak attacks in large language models (LLMs). We observe that existing defense mechanisms fail to generalize to harmful queries disguised by unseen attack templates, despite LLMs being capable of distinguishing disguised harmful queries in the embedding space. Based on these insights, we propose a two-stage defense approach: (i) pre-generation defense that detects harmful queries before response generation begins, and (ii) mid-generation defense that monitors partial responses during generation to prevent outputting more harmful content. Our MetaDefense trains the LLM to predict the harmfulness of both queries and partial responses using specialized prompts, enabling early termination of potentially harmful interactions. Extensive experiments across multiple LLM architectures (LLaMA-2-7B, Qwen-2.5-3B-Instruct, and LLaMA-3.2-3B-Instruct) demonstrate that MetaDefense significantly outperforms existing defense mechanisms, achieving robust defense against harmful queries with seen and unseen attack templates while maintaining competitive performance on benign tasks. Code is available at https://github.com/ws-jiang/MetaDefense.

Figures (5)

• Figure 1: ASR of harmful queries with direct and three unseen attack templates on LLaMA-2-7B.
• Figure 2: t-sne visualization of harmful and benign query embeddings. Best viewed in color.
• Figure 3: Harmful probability of GSM8K and harmful queries predicted by pre-generation defense.
• Figure 4: ASR, inference time per harmful query and GSM8K query with different $\gamma$'s.
• Figure 5: Harmful probability of benign and harmful queries (with different attack templates) predicted by the pre-generation defense.