Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Overlooked Safety Vulnerability in LLMs: Malicious Intelligent Optimization Algorithm Request and its Jailbreak

Haoran Gu, Handing Wang, Yi Mei, Mengjie Zhang, Yaochu Jin

TL;DR

This work identifies a previously overlooked safety risk in LLMs: their potential to assist in automatically designing malicious intelligent optimization algorithms. It introduces MalOptBench, a benchmark of 60 malicious optimization prompts across four tasks, and MOBjailbreak, a tailored jailbreak that rewrites prompts to bypass safeguards. Evaluating 13 mainstream LLMs, the study finds high vulnerability to malicious optimization requests under standard prompts and near-complete defeat of defenses under MOBjailbreak, highlighting the need for stronger alignment. The results suggest current safety defenses are insufficient in this domain, motivating continued research into robust, domain-aware safeguards for LLMs in algorithm design contexts. The authors also release MalOptBench to spur defensive developments while emphasizing ethical use and limitations of the study.

Abstract

The widespread deployment of large language models (LLMs) has raised growing concerns about their misuse risks and associated safety issues. While prior studies have examined the safety of LLMs in general usage, code generation, and agent-based applications, their vulnerabilities in automated algorithm design remain underexplored. To fill this gap, this study investigates this overlooked safety vulnerability, with a particular focus on intelligent optimization algorithm design, given its prevalent use in complex decision-making scenarios. We introduce MalOptBench, a benchmark consisting of 60 malicious optimization algorithm requests, and propose MOBjailbreak, a jailbreak method tailored for this scenario. Through extensive evaluation of 13 mainstream LLMs including the latest GPT-5 and DeepSeek-V3.1, we reveal that most models remain highly susceptible to such attacks, with an average attack success rate of 83.59% and an average harmfulness score of 4.28 out of 5 on original harmful prompts, and near-complete failure under MOBjailbreak. Furthermore, we assess state-of-the-art plug-and-play defenses that can be applied to closed-source models, and find that they are only marginally effective against MOBjailbreak and prone to exaggerated safety behaviors. These findings highlight the urgent need for stronger alignment techniques to safeguard LLMs against misuse in algorithm design.

Overlooked Safety Vulnerability in LLMs: Malicious Intelligent Optimization Algorithm Request and its Jailbreak

TL;DR

This work identifies a previously overlooked safety risk in LLMs: their potential to assist in automatically designing malicious intelligent optimization algorithms. It introduces MalOptBench, a benchmark of 60 malicious optimization prompts across four tasks, and MOBjailbreak, a tailored jailbreak that rewrites prompts to bypass safeguards. Evaluating 13 mainstream LLMs, the study finds high vulnerability to malicious optimization requests under standard prompts and near-complete defeat of defenses under MOBjailbreak, highlighting the need for stronger alignment. The results suggest current safety defenses are insufficient in this domain, motivating continued research into robust, domain-aware safeguards for LLMs in algorithm design contexts. The authors also release MalOptBench to spur defensive developments while emphasizing ethical use and limitations of the study.

Abstract

The widespread deployment of large language models (LLMs) has raised growing concerns about their misuse risks and associated safety issues. While prior studies have examined the safety of LLMs in general usage, code generation, and agent-based applications, their vulnerabilities in automated algorithm design remain underexplored. To fill this gap, this study investigates this overlooked safety vulnerability, with a particular focus on intelligent optimization algorithm design, given its prevalent use in complex decision-making scenarios. We introduce MalOptBench, a benchmark consisting of 60 malicious optimization algorithm requests, and propose MOBjailbreak, a jailbreak method tailored for this scenario. Through extensive evaluation of 13 mainstream LLMs including the latest GPT-5 and DeepSeek-V3.1, we reveal that most models remain highly susceptible to such attacks, with an average attack success rate of 83.59% and an average harmfulness score of 4.28 out of 5 on original harmful prompts, and near-complete failure under MOBjailbreak. Furthermore, we assess state-of-the-art plug-and-play defenses that can be applied to closed-source models, and find that they are only marginally effective against MOBjailbreak and prone to exaggerated safety behaviors. These findings highlight the urgent need for stronger alignment techniques to safeguard LLMs against misuse in algorithm design.
Paper Structure (22 sections, 1 equation, 5 figures, 4 tables)

This paper contains 22 sections, 1 equation, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Safety Vulnerability in Large Language Models
  4. LLM-based Intelligent Optimization Algorithm Design
  5. Motivation
  6. The MalOptBench Benchmark and Its Constructed Jailbreak Prompts
  7. Benchmark Construction
  8. Overview
  9. Adversarial User Generator and Task Rewriter
  10. Prompt Jailbreaks
  11. LLMs Adopted in Our Work
  12. Experimental Design
  13. Model
  14. Evaluation Metrics
  15. Results
  16. ...and 7 more sections

Figures (5)

  • Figure 1: An example of a malicious intelligent optimization algorithm design request and the responses given by LLMs.
  • Figure 2: Automated construction framework of MalOptBench.
  • Figure 3: An example of MOBjailbreak.
  • Figure 4: Token attention distribution for a malicious input that successfully triggers attacks against Gemma-2-9b-it.
  • Figure 5: Some prompt-response examples from MalOptBench against GPT-4o, Gemini-2.5-Flash, and DeepSeek-V3.1.