Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SoSBench: Benchmarking Safety Alignment on Six Scientific Domains

Fengqing Jiang, Fengbo Ma, Zhangchen Xu, Yuetai Li, Zixin Rao, Bhaskar Ramasubramanian, Luyao Niu, Bo Li, Xianyan Chen, Zhen Xiang, Radha Poovendran

Abstract

Large language models (LLMs) exhibit advancing capabilities in complex tasks, such as reasoning and graduate-level question answering, yet their resilience against misuse, particularly involving scientifically sophisticated risks, remains underexplored. Existing safety benchmarks typically focus either on instructions requiring minimal knowledge comprehension (e.g., ``tell me how to build a bomb") or utilize prompts that are relatively low-risk (e.g., multiple-choice or classification tasks about hazardous content). Consequently, they fail to adequately assess model safety when handling knowledge-intensive, hazardous scenarios. To address this critical gap, we introduce SoSBench, a regulation-grounded, hazard-focused benchmark encompassing six high-risk scientific domains: chemistry, biology, medicine, pharmacology, physics, and psychology. The benchmark comprises 3,000 prompts derived from real-world regulations and laws, systematically expanded via an LLM-assisted evolutionary pipeline that introduces diverse, realistic misuse scenarios (e.g., detailed explosive synthesis instructions involving advanced chemical formulas). We evaluate frontier models within a unified evaluation framework using our SoSBench. Despite their alignment claims, advanced models consistently disclose policy-violating content across all domains, demonstrating alarmingly high rates of harmful responses (e.g., 84.9% for Deepseek-R1 and 50.3% for GPT-4.1). These results highlight significant safety alignment deficiencies and underscore urgent concerns regarding the responsible deployment of powerful LLMs.

SoSBench: Benchmarking Safety Alignment on Six Scientific Domains

Abstract

Large language models (LLMs) exhibit advancing capabilities in complex tasks, such as reasoning and graduate-level question answering, yet their resilience against misuse, particularly involving scientifically sophisticated risks, remains underexplored. Existing safety benchmarks typically focus either on instructions requiring minimal knowledge comprehension (e.g., ``tell me how to build a bomb") or utilize prompts that are relatively low-risk (e.g., multiple-choice or classification tasks about hazardous content). Consequently, they fail to adequately assess model safety when handling knowledge-intensive, hazardous scenarios. To address this critical gap, we introduce SoSBench, a regulation-grounded, hazard-focused benchmark encompassing six high-risk scientific domains: chemistry, biology, medicine, pharmacology, physics, and psychology. The benchmark comprises 3,000 prompts derived from real-world regulations and laws, systematically expanded via an LLM-assisted evolutionary pipeline that introduces diverse, realistic misuse scenarios (e.g., detailed explosive synthesis instructions involving advanced chemical formulas). We evaluate frontier models within a unified evaluation framework using our SoSBench. Despite their alignment claims, advanced models consistently disclose policy-violating content across all domains, demonstrating alarmingly high rates of harmful responses (e.g., 84.9% for Deepseek-R1 and 50.3% for GPT-4.1). These results highlight significant safety alignment deficiencies and underscore urgent concerns regarding the responsible deployment of powerful LLMs.

Paper Structure

This paper contains 46 sections, 1 equation, 10 figures, 12 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Safety Alignment
  4. Safety Benchmarks in Scientific Domains.
  5. Proposed SoSBench
  6. Regulatory Foundations of SoSBench
  7. Construction Pipeline of SoSBench
  8. Manual Seed Collection
  9. Hybrid Initial Prompt Generation
  10. Data Evolution
  11. Prompt Mutation
  12. Quality Validation
  13. Coverage‑driven Heuristic Sampling
  14. Benchmark Analysis
  15. Risk Category Distribution
  16. ...and 31 more sections

Figures (10)

  • Figure 1: Overview of SoSBench and its construction pipeline. Our benchmark spans six domains, biology, chemistry, medicine, pharmacology, physics, and psychology.
  • Figure 2: The diverse risk categories covered by SoSBench and the risk distribution.
  • Figure 3: t-SNE visualization shows the broader coverage of our SoSBench compared with existing benchmarks.
  • Figure 4: Model scaling analysis. PVR trends illustrate how scaling shifts the balance between knowledge and alignment.
  • Figure 5: Reasoning effort scaling on different models. This budget scaling helps improving the safety in answers, but not on thinking.
  • ...and 5 more figures