Toward Reliable, Safe, and Secure LLMs for Scientific Applications

Abstract

As large language models (LLMs) evolve into autonomous "AI scientists," they promise transformative advances but introduce novel vulnerabilities, from potential "biosafety risks" to "dangerous explosions." Ensuring trustworthy deployment in science requires a new paradigm centered on reliability (ensuring factual accuracy and reproducibility), safety (preventing unintentional physical or biological harm), and security (preventing malicious misuse). Existing general-purpose safety benchmarks are poorly suited for this purpose, suffering from a fundamental domain mismatch, limited threat coverage of science-specific vectors, and benchmark overfitting, which create a critical gap in vulnerability evaluation for scientific applications. This paper examines the unique security and safety landscape of LLM agents in science. We begin by synthesizing a detailed taxonomy of LLM threats contextualized for scientific research, to better understand the unique risks associated with LLMs in science. Next, we conceptualize a mechanism to address the evaluation gap by utilizing dedicated multi-agent systems for the automated generation of domain-specific adversarial security benchmarks. Based on our analysis, we outline how existing safety methods can be brought together and integrated into a conceptual multilayered defense framework designed to combine a red-teaming exercise and external boundary controls with a proactive internal Safety LLM Agent. Together, these conceptual elements provide a necessary structure for defining, evaluating, and creating comprehensive defense strategies for trustworthy LLM agent deployment in scientific disciplines.

Figures (5)

• Figure 1: Demonstration of Ethical Compliance Evasion (jailbreak-style) user inputs across three scientific domains, chemical science, biology science, and infrastructure resilience, evaluated on three LLM agents (GPT-3.5, Claude 3.7, and Gemini 2.5 Pro). Each user input is designed as a red team scenario to probe model robustness against domain-specific unsafe or dual-use instructions. The red-colored text highlights potentially harmful content. Note: The "User Input" text shown represents a conceptual demonstration snippet rather than the complete adversarial prompt. Full prompt structures, which include complex role-playing wrappers, have been abbreviated to adhere to responsible disclosure and safety protocols. Complete prompts can be made available upon request for verification purposes.
• Figure 2: LLM threats taxonomy covering inference-time and training-time attack categories.
• Figure 3: Motivations & Risks of LLM Attacks in the Scientific Research Pipeline.
• Figure 4: Conceptual Multi-Agent Framework for Vulnerability Benchmark Generation for High-Stakes Scientific Applications. This model illustrates how specialized agents could collaborate to create and refine adversarial prompts. The conceptual workflow includes assigning specialized domain and adversarial roles, generating candidate prompts, and iteratively improving clarity and subtlety, as well as a quality control phase to filter redundancy, test guardrail efficacy, and ensure robustness. Such a system, with optional human-in-the-loop oversight, is envisioned to produce a high-quality, domain-specific benchmark dataset.
• Figure 5: Conceptual Defense Architecture for Multi-agent LLMs, Enhancing Reliability, Safety, and Security. The diagram illustrates the flow from user prompt through the external and internal Safety layers to produce a trusted response.