Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Resisting Correction: How RLHF Makes Language Models Ignore External Safety Signals in Natural Conversation

Felipe Biava Cataneo

TL;DR

The paper addresses the problem that external safety signals used to correct language models during inference often fail to influence model outputs in natural conversation, especially after RLHF-based instruction-tuning. It uses a causal-intervention protocol on Llama-3.2-3B with GSM8K to compare base and instruction-tuned models across four prompting strategies, revealing a strong context dependence: high controllability under explicit commands but severe resistance in natural dialogue, despite some internal calibration benefits. The key findings show $\rho \approx 1.0$ in command mode versus $\rho \approx 0.04$ and a bias of $+40\%$ in natural queries for the instruct-tuned model, along with $r = 0.035$ for internal calibration signals. The practical impact is that safety corrections in real-world chats require architectural overrides and mode-aware calibration, as natural-language prompts alone are insufficient for enforcing external safety signals in RLHF-tuned systems.

Abstract

Safety architectures for language models increasingly rely on external monitors to detect errors and inject corrective signals at inference time. For such systems to function in interactive settings, models must be able to incorporate externally provided confidence information into their verbal responses. In this work, we test whether instruction-tuned language models preserve this controllability across different interaction modes. Using Llama-3.2-3B on GSM8K, we perform a causal intervention study in which explicit external confidence signals are injected and model compliance is measured under multiple prompt strategies. We find that base models exhibit near-perfect controllability (Spearman rho close to 1.0), while instruction-tuned models display a striking context dependence: they fully comply with external corrections under explicit command prompts (bias approximately 0 percent, rho = 0.93), yet systematically ignore the same signals in natural conversational queries (bias plus 40 percent, rho = 0.04). This behavior is not a capability failure; the model can process the signal, but an emergent property of RLHF optimization that prioritizes conversational fluency over external calibration cues in natural dialogue. We further show that internal token-level confidence in small models is uninformative (r = 0.035), underscoring the necessity of external supervision. Our findings highlight a deployment-critical failure mode: the interaction style users expect is precisely where safety corrections are least effective.

Resisting Correction: How RLHF Makes Language Models Ignore External Safety Signals in Natural Conversation

TL;DR

The paper addresses the problem that external safety signals used to correct language models during inference often fail to influence model outputs in natural conversation, especially after RLHF-based instruction-tuning. It uses a causal-intervention protocol on Llama-3.2-3B with GSM8K to compare base and instruction-tuned models across four prompting strategies, revealing a strong context dependence: high controllability under explicit commands but severe resistance in natural dialogue, despite some internal calibration benefits. The key findings show in command mode versus and a bias of in natural queries for the instruct-tuned model, along with for internal calibration signals. The practical impact is that safety corrections in real-world chats require architectural overrides and mode-aware calibration, as natural-language prompts alone are insufficient for enforcing external safety signals in RLHF-tuned systems.

Abstract

Safety architectures for language models increasingly rely on external monitors to detect errors and inject corrective signals at inference time. For such systems to function in interactive settings, models must be able to incorporate externally provided confidence information into their verbal responses. In this work, we test whether instruction-tuned language models preserve this controllability across different interaction modes. Using Llama-3.2-3B on GSM8K, we perform a causal intervention study in which explicit external confidence signals are injected and model compliance is measured under multiple prompt strategies. We find that base models exhibit near-perfect controllability (Spearman rho close to 1.0), while instruction-tuned models display a striking context dependence: they fully comply with external corrections under explicit command prompts (bias approximately 0 percent, rho = 0.93), yet systematically ignore the same signals in natural conversational queries (bias plus 40 percent, rho = 0.04). This behavior is not a capability failure; the model can process the signal, but an emergent property of RLHF optimization that prioritizes conversational fluency over external calibration cues in natural dialogue. We further show that internal token-level confidence in small models is uninformative (r = 0.035), underscoring the necessity of external supervision. Our findings highlight a deployment-critical failure mode: the interaction style users expect is precisely where safety corrections are least effective.
Paper Structure (24 sections, 3 figures, 2 tables)

This paper contains 24 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. The Calibration Crisis in Small Language Models
  3. Motivating Scenario: Medical Diagnosis
  4. Research Questions
  5. Related Work
  6. Calibration and Uncertainty
  7. Modular AI Safety
  8. RLHF, Refusal, and Sycophancy
  9. Methodology
  10. Experimental Design
  11. Dataset and Models
  12. Procedure
  13. Prompt Strategies
  14. Results
  15. Result 1: Internal Calibration Failure
  16. ...and 9 more sections

Figures (3)

  • Figure 1: Context-Dependent Resistance. Left: In Command contexts, both models comply ($\rho \approx 1.0$). Right: In Natural contexts, Instruction-tuned models (red) ignore low-confidence hints, reporting high confidence ($\rho=0.036$), while Base model maintains compliance.
  • Figure 2: Strategy-Level Compliance. Left: Spearman correlation ($\rho$) shows Base model maintains high controllability across strategies, while Instruct model exhibits collapse specifically in natural_query (black border). Right: Bias analysis confirms the resistance is deployment-specific.
  • Figure 3: Calibration Curves. RLHF (right) improves intrinsic calibration compared to Base (left), reducing Expected Calibration Error. However, this improvement comes at the cost of reduced responsiveness to external corrections in conversational settings.