Noise Injection Systemically Degrades Large Language Model Safety Guardrails

TL;DR

This study examines the robustness of safety guardrails in open-weight LLMs by injecting zero-mean Gaussian noise into activations during inference, revealing systemic brittleness of post-hoc safety fine-tuning across multiple model families. The authors conduct three experiments (guardrail robustness, math-reasoning under noise, and noise-based safety-tuning) using datasets such as the harmful-instruction corpus and GSM8K, and evaluate with automated safety classifiers. Key findings show that activation noise increases harmful outputs across models (with $p<0.001$ and large effect sizes), that deeper safety fine-tuning does not reliably improve resilience, and that RL-based post-training (as in Qwen2.5) yields notably stronger robustness. Noise-tuned safety training offers localized improvements within training noise ranges but does not generalize to higher perturbations, underscoring the need for deeper, more integrated safety mechanisms that resist internal perturbations in real-world deployment.

Abstract

Safety guardrails in large language models (LLMs) are a critical component in preventing harmful outputs. Yet, their resilience under perturbation remains poorly understood. In this paper, we investigate the robustness of safety fine-tuning in LLMs by systematically injecting Gaussian noise into model activations. We show across multiple open-weight models that (1) Gaussian noise raises harmful-output rates (p < 0.001) by up to 27%, (2) that deeper safety fine-tuning affords no extra protection, and (3) that chain-of-thought reasoning remains largely intact. The findings reveal critical vulnerabilities in current safety alignment techniques and highlight the potential of reasoning-based and reinforcement learning approaches as promising direction for developing more robust AI safety systems. These results have important implications for real-world deployment of LLMs in safety-critical applications as these results imply that widely-deployed safety tuning methods can fail even without adversarial prompts.

Figures (8)

• Figure 1: a) When injecting Gaussian noise into all model activations layers, we see a significant increase in harmful output generation in open-weight LLMs, revealing the brittleness of safety fine-tuning to internal perturbations. b) Similarly, we found that across varying fine-tuning depths, models converge to similar performance under high noise, highlighting that deeper fine-tuning does not necessarily improve robustness to noise injection. However, while noise-tuned safety training improves robustness within specific noise ranges, this protection does not generalize to higher noise values.
• Figure 2: Performance of various models under different noise levels against noise-injection attacks (lower is better). Noise injection into model activations during inference significantly increased harmful response outputs in most tested models ($p < 0.001, n=12)$, with large effect sizes (Cohen's d ranging from -1.26 to -9.29).
• Figure 3: Performance under noise degradation for each fine-tuned model variant. The left figure has fine-tuned models with varying unique training samples used for a fixed dataset size of 7000. The right figure has fine-tuned models with varying training epochs for a training set of 7000 unique samples. In both cases, the final model performances are within the same range, indicating that more fine-tuning doesn't lead to higher resilience to noise. Additional plots can be found in Appendix I for the other combinations of training epochs versus unique training samples.
• Figure 4: Performance of base and noise-tuned Gemma 2B models under varying noise levels against noise-injection attacks (lower is better). Noise-tuned training increases robustness within the training noise ranges but provides limited protection against higher noise values, as shown in the high-noise results.
• Figure 5: The default prompt template for Meta Llama Guard 3, used to evaluate the safety_score of a given completion.