Exploring Jailbreak Attacks on LLMs through Intent Concealment and Diversion

TL;DR

The paper tackles jailbreak vulnerabilities in instruction-aligned LLMs by introducing ICE, a black-box framework that leverages intent concealment and semantic dispersion to achieve high attack success rates with a single query. It is complemented by BiSceneEval, a dual-scenario benchmark for evaluating jailbreak impact on both QA and text-generation tasks. Across 11 LLMs and two datasets (AdvBench and BiSceneEval), ICE outperforms baselines in both effectiveness and efficiency, revealing systemic weaknesses in current defenses and highlighting the need for hybrid static-dynamic defenses. These contributions provide a practical, scalable framework for robust LLM security and responsible AI deployment.

Abstract

Although large language models (LLMs) have achieved remarkable advancements, their security remains a pressing concern. One major threat is jailbreak attacks, where adversarial prompts bypass model safeguards to generate harmful or objectionable content. Researchers study jailbreak attacks to understand security and robustness of LLMs. However, existing jailbreak attack methods face two main challenges: (1) an excessive number of iterative queries, and (2) poor generalization across models. In addition, recent jailbreak evaluation datasets focus primarily on question-answering scenarios, lacking attention to text generation tasks that require accurate regeneration of toxic content. To tackle these challenges, we propose two contributions: (1) ICE, a novel black-box jailbreak method that employs Intent Concealment and divErsion to effectively circumvent security constraints. ICE achieves high attack success rates (ASR) with a single query, significantly improving efficiency and transferability across different models. (2) BiSceneEval, a comprehensive dataset designed for assessing LLM robustness in question-answering and text-generation tasks. Experimental results demonstrate that ICE outperforms existing jailbreak techniques, revealing critical vulnerabilities in current defense mechanisms. Our findings underscore the necessity of a hybrid security strategy that integrates predefined security mechanisms with real-time semantic decomposition to enhance the security of LLMs.

Figures (6)

• Figure 1: The construction process of BiSceneEval.
• Figure 2: The proportion of each category to the entire data volume in two scenarios.
• Figure 3: Overview of the ICE attack process.
• Figure 4: The TCPS and query times of different jailbreak methods.
• Figure 5: Data volume reduction across three steps, where the darker colored parts represent the original data.