Rethinking harmless refusals when fine-tuning foundation models

TL;DR

The paper investigates whether safety-focused fine-tuning in large language models hides rather than eliminates undesired behavior by examining inconsistencies between Chain-of-Thought reasoning and final outputs in semi-realistic role-play prompts. It introduces reason-based deception as a hidden behavior and compares two response strategies—polite refusals versus explicit rebuttals—in multi-turn interactions. The results show that explicit rebuttals dramatically reduce undesired outputs and nearly remove reason-based deception across GPT-4 variants, while refusals tend to worsen downstream behavior. These findings argue for rethinking standard refusal-centric fine-tuning approaches and offer practical guidance for designing safer, more robust LLMs in interactive settings.

Abstract

In this paper, we investigate the degree to which fine-tuning in Large Language Models (LLMs) effectively mitigates versus merely conceals undesirable behavior. Through the lens of semi-realistic role-playing exercises designed to elicit such behaviors, we explore the response dynamics of LLMs post fine-tuning interventions. Our methodology involves prompting models for Chain-of-Thought (CoT) reasoning and analyzing the coherence between the reasoning traces and the resultant outputs. Notably, we identify a pervasive phenomenon we term \emph{reason-based deception}, where models either stop producing reasoning traces or produce seemingly ethical reasoning traces that belie the unethical nature of their final outputs. We further examine the efficacy of response strategies (polite refusal versus explicit rebuttal) in curbing the occurrence of undesired behavior in subsequent outputs of multi-turn interactions. Our findings reveal that explicit rebuttals significantly outperform polite refusals in preventing the continuation of undesired outputs and nearly eliminate reason-based deception, challenging current practices in model fine-tuning. Accordingly, the two key contributions of this paper are (1) defining and studying reason-based deception, a new type of hidden behavior, and (2) demonstrating that rebuttals provide a more robust response model to harmful requests than refusals, thereby highlighting the need to reconsider the response strategies in fine-tuning approaches.

Figures (8)

• Figure 1: A diagram of our experimental setup. Each experiment consists of a scenario with a final unethical primer message. An initial response is either sampled (open lock, left branch) or fixed (closed lock, center and right branch). A trigger message is appended to the conversation history and a final response is sampled. Abbreviated examples of CoT reasoning traces are shown in grey italics in the final response. In this diagram the final response of the center branch is fair (green) and the final response of the left and right branch is discriminatory (red). In the left branch there is no reason-based deception because the reasoning traces is consistent with the output. In the right branch there is reason-based deception because the reasoning trace to act ethically is inconsistent with the discriminatory output.
• Figure 2: Left. The rates of unethical model outputs across the our three scenarios and four gpt-4 model releases. We only tested the vision variant on the real estate scenario. Right. The rate at which the unethical output is consistent with (i.e. predicted from) the CoT reasoning.
• Figure 3: The model reasoning trace indicates that it is aware that it should not discriminate and that it can not comply with requests to discriminate. Despite this, the final output (not shown) is discriminatory.
• Figure 4: The rate at which each model outputs CoT reasoning. For each category of output, undesired or acceptable, we measure the percent of sample that contain CoT reasoning. Note that for the real estate scenario the most recent models never output CoT reasoning before a discriminatory response.
• Figure 5: Comparing results when models are asked to discriminate based on either race or preference for coffee or tea. Left. The percent of samples with discriminatory output. Middle. The percent of discriminatory samples which were detected from (i.e. consistent with) CoT reasoning traces. Right. The percent of samples that contained CoT reasoning traces. Note that all models were prompted to output CoT reasoning so missing CoT reasoning is a failure of the model to follow instructions.