EquaCode: A Multi-Strategy Jailbreak Approach for Large Language Models via Equation Solving and Code Completion

TL;DR

EquaCode targets LLM safety by introducing a multi-strategy jailbreak that converts malicious prompts into a two-step process: first solve a symbolic equation $\mathsf{B}+\mathsf{C}+x=\mathsf{A}$ to obtain execution steps $x$, then complete these steps via a structured Python Solver class using $\mathsf{A}$, $\mathsf{B}$, and $\mathsf{C}$. This cross-domain approach yields a synergistic performance greater than either the equation or code module alone, with high attack success rates across GPT-series models (e.g., up to $98.46\%$ on GPT-4-Turbo) and strong results on other state-of-the-art LLMs. Ablation studies and gradient-based saliency analyses support the claim of a synergistic effect driven by misalignment between safety filters and capabilities in mathematics and coding. The work also discusses defense strategies and limitations, highlighting the need for diversified defenses and future work to automate and broaden attack combinations. Overall, EquaCode reveals a cross-domain vulnerability amplification in LLM safety and offers a framework for evaluating and improving jailbreak resistance.

Abstract

Large language models (LLMs), such as ChatGPT, have achieved remarkable success across a wide range of fields. However, their trustworthiness remains a significant concern, as they are still susceptible to jailbreak attacks aimed at eliciting inappropriate or harmful responses. However, existing jailbreak attacks mainly operate at the natural language level and rely on a single attack strategy, limiting their effectiveness in comprehensively assessing LLM robustness. In this paper, we propose Equacode, a novel multi-strategy jailbreak approach for large language models via equation-solving and code completion. This approach transforms malicious intent into a mathematical problem and then requires the LLM to solve it using code, leveraging the complexity of cross-domain tasks to divert the model's focus toward task completion rather than safety constraints. Experimental results show that Equacode achieves an average success rate of 91.19% on the GPT series and 98.65% across 3 state-of-the-art LLMs, all with only a single query. Further, ablation experiments demonstrate that EquaCode outperforms either the mathematical equation module or the code module alone. This suggests a strong synergistic effect, thereby demonstrating that multi-strategy approach yields results greater than the sum of its parts.

Figures (3)

• Figure 1: Overview of the EquaCode approach, which consists of two modules: (1) Equation Module – This module utilizes mathematical symbols to transform the malicious query into an equation by associating three components: subject, tool, and steps. (2) Code Module – This module embeds the equation’s components along with the malicious query into a wrapped Solver class, requiring the LLM to complete the execution steps and describe the malicious tools. Through this integrated attack approach, LLMs are induced to complete harmful procedures within the solve function.
• Figure 2: Attack module based on code encapsulation
• Figure 3: intention