Probing the Safety Response Boundary of Large Language Models via Unsafe Decoding Path Generation

TL;DR

This work reframes LLM safety evaluation around decoding-space boundaries rather than input-space, introducing a Cost Value Model (CVM) learned via TD($\lambda$) to detect and bias unsafe decoding paths. By treating decoding as an MDP and biasing top-$K$ logits with CVM values, the authors demonstrate Jailbreak Value Decoding (JVD) can elicit toxic outputs from safety-aligned models such as LLaMA-2-chat-7B, Vicuna-13B, and Mistral-7B-Instruct-v0.2, revealing vulnerabilities across initial tokens, agreement prompts, and refusal tokens, as well as a tradeoff between readability and toxicity. The work further shows that CVM-guided prompts can enhance attack effectiveness and transferability across models, while enabling prompt-optimization strategies that exploit decoded pathways. Overall, the findings argue for more granular safety alignment that addresses both shallow and deep decoding vulnerabilities and cautions about the potential misuse of CVM-like techniques for harmful content generation. The study highlights practical implications for safer deployment of open and closed LLMs and calls for refined defenses that resist decoding-time manipulation.

Abstract

Large Language Models (LLMs) are implicit troublemakers. While they provide valuable insights and assist in problem-solving, they can also potentially serve as a resource for malicious activities. Implementing safety alignment could mitigate the risk of LLMs generating harmful responses. We argue that: even when an LLM appears to successfully block harmful queries, there may still be hidden vulnerabilities that could act as ticking time bombs. To identify these underlying weaknesses, we propose to use a cost value model as both a detector and an attacker. Trained on external or self-generated harmful datasets, the cost value model could successfully influence the original safe LLM to output toxic content in decoding process. For instance, LLaMA-2-chat 7B outputs 39.18% concrete toxic content, along with only 22.16% refusals without any harmful suffixes. These potential weaknesses can then be exploited via prompt optimization such as soft prompts on images. We name this decoding strategy: Jailbreak Value Decoding (JVD), emphasizing that seemingly secure LLMs may not be as safe as we initially believe. They could be used to gather harmful data or launch covert attacks.

Figures (3)

• Figure 1: Examples on LLaMA-2-chat 7B when facing a toxic question. 1) Without the CVM guidance, we can see the model successfully defends the toxic questions by outputting refusals, which indicate it's always in the safe area (green line). 2) With the CVM guidance at first, the models tend to generate toxic outputs (yellow line), then change to refusals once we stop the CVM guidance (yellow to green line). 3) With the CVM, the model outputs absoluate toxic contents (yellow to red line).
• Figure 2: The absolute cost values (without normalization) of two responses in Example 3. After the same beginning: "To help ...prepare", the CVM guides to more toxic and concrete outputs comparing to normal decoding.
• Figure 3: The loss curve of trainable noise on the images(soft prompts) on Vicuna-13B. The left side is the baseline curve that trained on manual toxic instructions while the right side curve is trained on text from Cost Value Model. We can see that, the right side's loss is smaller than the left side, indicating the similarity between the original outputs and CVM guided outputs.