ROSE Doesn't Do That: Boosting the Safety of Instruction-Tuned Large Language Models with Reverse Prompt Contrastive Decoding

TL;DR

The paper tackles safety in instruction-tuned LLMs by proposing Reverse Prompt Contrastive Decoding (ROSE), an inference-time, training-free method that suppresses undesired outputs induced by carefully crafted reverse prompts to boost safe behavior. ROSE uses a contrastive decoding objective that reduces the scoring of unsafe tokens when conditioned on a reverse prompt, thereby biasing generation toward safe responses. Empirical evaluations across six safety benchmarks and two general-purpose tasks over five model families show ROSE delivers consistent safety gains up to about +14% and can also enhance general-purpose performance, with robustness across prompt variants and hyperparameters. The method is plug-and-play, compatible with existing safety-tuned approaches, and provides a scalable route to safer deployment of instruction-tuned LLMs.

Abstract

With the development of instruction-tuned large language models (LLMs), improving the safety of LLMs has become more critical. However, the current approaches for aligning the LLMs output with expected safety usually require substantial training efforts, e.g., high-quality safety data and expensive computational resources, which are costly and inefficient. To this end, we present reverse prompt contrastive decoding (ROSE), a simple-yet-effective method to directly boost the safety of existing instruction-tuned LLMs without any additional training. The principle of ROSE is to improve the probability of desired safe output via suppressing the undesired output induced by the carefully-designed reverse prompts. Experiments on 6 safety and 2 general-purpose tasks show that, our ROSE not only brings consistent and significant safety improvements (up to +13.8% safety score) upon 5 types of instruction-tuned LLMs, but also benefits the general-purpose ability of LLMs. In-depth analyses explore the underlying mechanism of ROSE, and reveal when and where to use it.

Figures (11)

• Figure 1: Illustration of Rose.Rose boosts the safety of LLMs by suppressing the undesired output induced by the reverse prompt. For ease of illustration, we only show the simplified prompts and logits in this figure.
• Figure 2: Performance comparison (%) of regular decoding v.s. our proposed Rose, with using the Alpaca-7B/13B as backbone models. "Reverse Prompt" means that we perform the regular decoding using the reverse prompt as the system prompt. The y-axis denotes the safety performance evaluated by ChatGPT for each task, where the evaluation details can be found in §\ref{['sec:experiments']} and the full results are in Table \ref{['tab:overall_safety_result']}. We see that Rose improves the safety over the regular decoding by a large margin (up to +13.98% score) across various safety datasets.
• Figure 3: Comparative winning rates (%) of Regular decoding (w/ sys. prompt) v.s. Ours ("Manual" prompt). We evaluate Alpaca-7b/13b and Vicuna-7b/13b models on (a) DangerousQA, (b) HarmfulQA, (c) Xstest (unsafe prompt) and (d) Do-Not-Answer benchmarks. Notably, we use the ChatGPT as the automated evaluator. It can be found that our Rose consistently outperforms the regular decoding among all models and benchmarks.
• Figure 4: Ablation study on reverse prompts. We evaluate the Alpaca-7B on the CValues and SafetyBench.
• Figure 5: Effect of $\alpha$. We show the safety score (on CValues) of Alpaca-7B using Rose across varied $\alpha$.