Are Smarter LLMs Safer? Exploring Safety-Reasoning Trade-offs in Prompting and Fine-Tuning

TL;DR

This work examines how strengthening LLM reasoning through prompting and fine-tuning affects safety and privacy. It reveals a consistent negative relationship between reasoning gains and safety when using prompt-based methods across multiple models and jailbreak attacks, and shows that fine-tuning on CoT-style and especially long CoT data can cause substantial safety degradation. The authors identify mitigation strategies, including careful prompting to balance safety and reasoning and incorporating reflective safety data during long CoT fine-tuning to achieve Pareto improvements. They extend the analysis to privacy, observing similar safety-privacy trade-offs, and propose practical safeguards for making reasoning-enhanced LLMs both more capable and trustworthy. Overall, the paper highlights the need for safety-aware design in reasoning-driven LLM improvements and points to directions like reinforced or multimodal reasoning to preserve trustworthiness.

Abstract

Large Language Models (LLMs) have demonstrated remarkable success across various NLP benchmarks. However, excelling in complex tasks that require nuanced reasoning and precise decision-making demands more than raw language proficiency--LLMs must reason, i.e., think logically, draw from past experiences, and synthesize information to reach conclusions and take action. To enhance reasoning abilities, approaches such as prompting and fine-tuning have been widely explored. While these methods have led to clear improvements in reasoning, their impact on LLM safety remains less understood. In this work, we investigate the interplay between reasoning and safety in LLMs. We highlight the latent safety risks that arise as reasoning capabilities improve, shedding light on previously overlooked vulnerabilities. At the same time, we explore how reasoning itself can be leveraged to enhance safety, uncovering potential mitigation strategies. By examining both the risks and opportunities in reasoning-driven LLM safety, our study provides valuable insights for developing models that are not only more capable but also more trustworthy in real-world deployments.

Figures (5)

• Figure 1: Safety challenges in improving LLM reasoning: Starting from the default Llama-2-7B-Chat, we experiment with improving its reasoning abilities via (1) zero-shot Chain-of-Thought (CoT) prompting, (2) few-shot CoT, (3) fine-tuning on a coding and math dataset, and (4) fine-tuning on the Open-o1-SFT dataset. As the accuracy increases from 16% to 41.4%, we see the model experiences a catastrophic drop of 66% in safety score, suggesting the existence of non-negligible safety challenges in improving LLM reasoning.
• Figure 2: The negative correlation between accuracy and safety. We evaluate seven LLMs with eleven prompts under three jailbreak attacks, GCG, AIR, and the decomposition attack (Decom.). Each dot stands for a prompt positioned by its averaged accuracy and safety across the models. We present the correlation coefficient (C) between accuracy and safety against each attack, all of which show strong negative correlation between the two indicators.
• Figure 3: Effects of different prompting strategies. Few-shot CoTs are denoted as -n, with -M and -S representing math and safety demonstrations, respectively. Sys-t1 and Sys-ref are reflection-based system prompts. ICD and Ref-S are safety-oriented prompts. (a) Using CoT prompts and adding math-specific CoT examples decreases safety compared to the default system prompt. (b) Reflection-based system prompts lower safety scores relative to the default prompt. (c) Four safety-oriented prompts significantly reduce GSM8K accuracy compared to a naive CoT prompt. (d) Extending the CoT prompt with safety and math demonstrations (with switched order) alleviates the trade-off between accuracy and safety.
• Figure 4: Long CoT fine-tuning can lead to catastrophic damage to safety. We fine-tune Llama-2-7B-Chat on two open-source long CoT datasets, O1-OPEN/OpenO1-SFT and NovaSky-AI/Sky-T1_data_17k. We fine-tune Llama-2-7B-Chat on incrementally more percentages of the datasets and evaluate the direct safety scores. Although the two datasets exhibit different scaling trend as the amount of data increases, we observe both datasets causes decreases of at least 30% in safety score when considering the worse cases.
• Figure 5: Effects of inference-time prompting to privacy. We see that incorporating prompts to improve reasoning will also bring negative effects to privacy.