Do Methods to Jailbreak and Defend LLMs Generalize Across Languages?

TL;DR

The paper investigates cross-lingual generalization of jailbreaks and defenses for large language models across ten languages using HarmBench and AdvBench, applying two jailbreak modalities and a simple defense. It finds substantial variability in attack success and defense robustness by language and model, with high-resource languages sometimes safer on standard prompts yet more vulnerable to adversarial prompts. The results show that simple defenses like Self-Verifier prompting and multilingual classifiers can reduce unsafe outputs but require language- and model-specific tuning. The findings highlight the need for language-aware safety benchmarks and targeted defense strategies to ensure equitable LLM safety across languages.

Abstract

Large language models (LLMs) undergo safety alignment after training and tuning, yet recent work shows that safety can be bypassed through jailbreak attacks. While many jailbreaks and defenses exist, their cross-lingual generalization remains underexplored. This paper presents the first systematic multilingual evaluation of jailbreaks and defenses across ten languages -- spanning high-, medium-, and low-resource languages -- using six LLMs on HarmBench and AdvBench. We assess two jailbreak types: logical-expression-based and adversarial-prompt-based. For both types, attack success and defense robustness vary across languages: high-resource languages are safer under standard queries but more vulnerable to adversarial ones. Simple defenses can be effective, but are language- and model-dependent. These findings call for language-aware and cross-lingual safety benchmarks for LLMs.

Figures (9)

• Figure 1: A heat map of unsafe response rates of LLMs from GPT, Cluade, Qwen, and Llama families to standard queries from HarmBench for 10 languages, including high-resource languages such as Spanish, medium-resource languages such as Arabic, or low-resource languages such as Swahili (as ordered on x-axis).
• Figure 2: A heat map of unsafe response rates of LLMs from GPT, Cluade, Qwen, and Llama families to standard queries in AdvBench in 10 languages, including high-resource languages such as Spanish, medium-resource languages such as Arabic, or low-resource languages such as Swahili (as ordered on x-axis).
• Figure 3: Unsafe rates by category and languages of the responses to standard queries in HarmBench. Results are averaged over models.
• Figure 4: Safety rank of languages over models for each prompting strategy (standard queries, logic jailbreaking, or andriushchenko2025jailbreaking jailbreaking)
• Figure 5: Language ranking variations over models and promptings (standard queries, logic jailbreaking, or andriushchenko2025jailbreaking's jailbreaking.)