Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Safe Multilingual Frontier AI

Artūrs Kanepajs, Vladimir Ivanov, Richard Moulange

TL;DR

This work examines how a language's level of resourcing relates to how vulnerable LLMs are to multilingual jailbreaks in that language, and proposes policy actions that align with the EU legal landscape and institutional framework to address multilingual jailbreaks, while promoting linguistic inclusivity.

Abstract

Linguistically inclusive LLMs -- which maintain good performance regardless of the language with which they are prompted -- are necessary for the diffusion of AI benefits around the world. Multilingual jailbreaks that rely on language translation to evade safety measures undermine the safe and inclusive deployment of AI systems. We provide policy recommendations to enhance the multilingual capabilities of AI while mitigating the risks of multilingual jailbreaks. We examine how a language's level of resourcing relates to how vulnerable LLMs are to multilingual jailbreaks in that language. We do this by testing five advanced AI models across 24 official languages of the EU. Building on prior research, we propose policy actions that align with the EU legal landscape and institutional framework to address multilingual jailbreaks, while promoting linguistic inclusivity. These include mandatory assessments of multilingual capabilities and vulnerabilities, public opinion research, and state support for multilingual AI development. The measures aim to improve AI safety and functionality through EU policy initiatives, guiding the implementation of the EU AI Act and informing regulatory efforts of the European AI Office.

Towards Safe Multilingual Frontier AI

TL;DR

This work examines how a language's level of resourcing relates to how vulnerable LLMs are to multilingual jailbreaks in that language, and proposes policy actions that align with the EU legal landscape and institutional framework to address multilingual jailbreaks, while promoting linguistic inclusivity.

Abstract

Linguistically inclusive LLMs -- which maintain good performance regardless of the language with which they are prompted -- are necessary for the diffusion of AI benefits around the world. Multilingual jailbreaks that rely on language translation to evade safety measures undermine the safe and inclusive deployment of AI systems. We provide policy recommendations to enhance the multilingual capabilities of AI while mitigating the risks of multilingual jailbreaks. We examine how a language's level of resourcing relates to how vulnerable LLMs are to multilingual jailbreaks in that language. We do this by testing five advanced AI models across 24 official languages of the EU. Building on prior research, we propose policy actions that align with the EU legal landscape and institutional framework to address multilingual jailbreaks, while promoting linguistic inclusivity. These include mandatory assessments of multilingual capabilities and vulnerabilities, public opinion research, and state support for multilingual AI development. The measures aim to improve AI safety and functionality through EU policy initiatives, guiding the implementation of the EU AI Act and informing regulatory efforts of the European AI Office.
Paper Structure (22 sections, 1 equation, 12 figures, 5 tables)

This paper contains 22 sections, 1 equation, 12 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. What to measure?
  4. Risk and capability measurement
  5. Risk mitigation and capability improvements
  6. Who will do the evaluations?
  7. Comprehensive empirical assessments and policy recommendations
  8. Methodology for the quantitative analysis
  9. Case Study: The EU and its 24 official languages
  10. Legal aspects
  11. Who benefits from multilingual AI in the EU, and how?
  12. Who could implement the policies?
  13. Regression results
  14. Discussion
  15. Capability and Safety Requirements to Introduce
  16. ...and 7 more sections

Figures (12)

  • Figure 1: Threat model and policy opportunity. Neglecting multilingual jailbreaks or multilingual capabilities can increase risks or limit AI benefit diffusion. Conversely, appropriately addressing the risks as well as capabilities can bring the benefits of safe multilingual frontier AI.
  • Figure 2: Framework for involvement of stakeholders to produce policy requirements that lead to safe multilingual frontier AI.
  • Figure 3: Claude 3.5 Sonnet, Harmful Accepted Proportion (100 observations per language)
  • Figure 4: Claude 3.5 Sonnet, Harmless Rejected Proportion (100 observations per language)
  • Figure 5: Gemini 1.5 Pro, Harmful Accepted Proportion (100 observations per language)
  • ...and 7 more figures