Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Malicious Repurposing of Open Science Artefacts by Using Large Language Models

Zahra Hashemi, Zhiqiang Zhong, Jun Pang, Wei Zhao

TL;DR

This paper examines the dual-use risks of repurposing open science artefacts with large language models. It introduces an end-to-end pipeline that jailbreaks LLMs via persuasion, extracts misuse-prone assets from NLP papers, and generates structured malicious research proposals across a seven-stage process, followed by an AI-safety evaluation framework that rates proposals on harmfulness, feasibility, and technical soundness. Applying this workflow to 51 ACL papers with three state-of-the-art LLMs, the authors demonstrate that LLMs can produce technically plausible yet harmful proposals once safety is bypassed, while also showing substantial disagreement among LLM evaluators, underscoring the limitations of automated judgments and the necessity of human oversight. The work highlights broad dual-use risks, calls for robust defenses, and argues for combined human-AI evaluation to credibly assess misuse risks in open-science artefacts.

Abstract

The rapid evolution of large language models (LLMs) has fuelled enthusiasm about their role in advancing scientific discovery, with studies exploring LLMs that autonomously generate and evaluate novel research ideas. However, little attention has been given to the possibility that such models could be exploited to produce harmful research by repurposing open science artefacts for malicious ends. We fill the gap by introducing an end-to-end pipeline that first bypasses LLM safeguards through persuasion-based jailbreaking, then reinterprets NLP papers to identify and repurpose their artefacts (datasets, methods, and tools) by exploiting their vulnerabilities, and finally assesses the safety of these proposals using our evaluation framework across three dimensions: harmfulness, feasibility of misuse, and soundness of technicality. Overall, our findings demonstrate that LLMs can generate harmful proposals by repurposing ethically designed open artefacts; however, we find that LLMs acting as evaluators strongly disagree with one another on evaluation outcomes: GPT-4.1 assigns higher scores (indicating greater potential harms, higher soundness and feasibility of misuse), Gemini-2.5-pro is markedly stricter, and Grok-3 falls between these extremes. This indicates that LLMs cannot yet serve as reliable judges in a malicious evaluation setup, making human evaluation essential for credible dual-use risk assessment.

Malicious Repurposing of Open Science Artefacts by Using Large Language Models

TL;DR

This paper examines the dual-use risks of repurposing open science artefacts with large language models. It introduces an end-to-end pipeline that jailbreaks LLMs via persuasion, extracts misuse-prone assets from NLP papers, and generates structured malicious research proposals across a seven-stage process, followed by an AI-safety evaluation framework that rates proposals on harmfulness, feasibility, and technical soundness. Applying this workflow to 51 ACL papers with three state-of-the-art LLMs, the authors demonstrate that LLMs can produce technically plausible yet harmful proposals once safety is bypassed, while also showing substantial disagreement among LLM evaluators, underscoring the limitations of automated judgments and the necessity of human oversight. The work highlights broad dual-use risks, calls for robust defenses, and argues for combined human-AI evaluation to credibly assess misuse risks in open-science artefacts.

Abstract

The rapid evolution of large language models (LLMs) has fuelled enthusiasm about their role in advancing scientific discovery, with studies exploring LLMs that autonomously generate and evaluate novel research ideas. However, little attention has been given to the possibility that such models could be exploited to produce harmful research by repurposing open science artefacts for malicious ends. We fill the gap by introducing an end-to-end pipeline that first bypasses LLM safeguards through persuasion-based jailbreaking, then reinterprets NLP papers to identify and repurpose their artefacts (datasets, methods, and tools) by exploiting their vulnerabilities, and finally assesses the safety of these proposals using our evaluation framework across three dimensions: harmfulness, feasibility of misuse, and soundness of technicality. Overall, our findings demonstrate that LLMs can generate harmful proposals by repurposing ethically designed open artefacts; however, we find that LLMs acting as evaluators strongly disagree with one another on evaluation outcomes: GPT-4.1 assigns higher scores (indicating greater potential harms, higher soundness and feasibility of misuse), Gemini-2.5-pro is markedly stricter, and Grok-3 falls between these extremes. This indicates that LLMs cannot yet serve as reliable judges in a malicious evaluation setup, making human evaluation essential for credible dual-use risk assessment.
Paper Structure (34 sections, 4 figures, 2 tables)

This paper contains 34 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Automated Scientific Research.
  4. Jailbreaking LLMs for Harmful Content.
  5. LLMs-as-Judge.
  6. Method
  7. Step 1: Jailbreaking LLMs via Persuasion Techniques
  8. Step 2: Extracting Malicious Research Questions
  9. Our Prompt Design.
  10. Step 3: Generating Step-wise Malicious Proposals
  11. Stage 1 (Problem Identification)
  12. Stage 2 (Literature Review)
  13. Stage 3 (Hypothesis Formation)
  14. Stage 4 (Experimental Design and Methodology)
  15. Stage 5 (Implementation Simulation)
  16. ...and 19 more sections

Figures (4)

  • Figure 1: Simplified pipeline for generating malicious research proposals and dual-use risk evaluation.
  • Figure 2: Seven-stage malicious proposal generation pipeline following the scientific method. Unlike traditional research that includes iterative troubleshooting and actual implementation, our approach simulates execution at Step 5 (highlighted), as running experiments is resource-intensive and beyond scope. Step 2 (highlighted) demonstrates API-based dataset selection, enabling access to a wider pool of resources than fixed libraries.
  • Figure 3: Radar charts presenting the average scores of proposals generated by GPT-4.1, Grok-3, and Gemini-2.5-pro.
  • Figure 4: Radar charts presenting the average evaluation scores assigned by GPT-4.1, Grok-3, and Gemini-2.5-pro when acting as evaluators.