DuoGuard: A Two-Player RL-Driven Framework for Multilingual LLM Guardrails

TL;DR

DuoGuard introduces a theoretically grounded two-player RL framework where a data generator and a guardrail classifier co-evolve to synthesize multilingual training data and improve guardrail performance. The authors prove convergence to a Nash equilibrium and demonstrate that the approach yields substantial multilingual safety gains, outperforming state-of-the-art baselines by over 20% on average while delivering up to 4.5x faster inference with a 0.5B classifier. They implement a practical iterative pipeline with DPO-based generator updates, seed-data curation, and targeted data filtering, achieving balanced multilingual coverage and mitigating low-resource language gaps. Empirically, DuoGuard shows strong cross-language generalization, enabling effective post-training data synthesis for multilingual toxicity detection across English, French, Spanish, and German, and is released alongside code and data for reproducibility.

Abstract

The rapid advancement of large language models (LLMs) has increased the need for guardrail models to ensure responsible use, particularly in detecting unsafe and illegal content. While substantial safety data exist in English, multilingual guardrail modeling remains underexplored due to the scarcity of open-source safety data in other languages. To address this gap, we propose a novel two-player Reinforcement Learning (RL) framework, where a generator and a guardrail model co-evolve adversarially to produce high-quality synthetic data for multilingual guardrail training. We theoretically formalize this interaction as a two-player game, proving convergence to a Nash equilibrium. Empirical evaluations show that our model \ours outperforms state-of-the-art models, achieving nearly 10% improvement over LlamaGuard3 (8B) on English benchmarks while being 4.5x faster at inference with a significantly smaller model (0.5B). We achieve substantial advancements in multilingual safety tasks, particularly in addressing the imbalance for lower-resource languages in a collected real dataset. Ablation studies emphasize the critical role of synthetic data generation in bridging the imbalance in open-source data between English and other languages. These findings establish a scalable and efficient approach to synthetic data generation, paving the way for improved multilingual guardrail models to enhance LLM safety. Code, model, and data will be open-sourced at https://github.com/yihedeng9/DuoGuard.

Figures (9)

• Figure 1: Illustration of the use-case of a guardrail model for LLMs, which functions as moderation between the user-LLM conversation.
• Figure 2: Overview of our main results. In the left figure, we demonstrate a consistently superior performance of average f1 score across 6 benchmarks in the four languages. In the right figure, we show that our model maintains the lowest inference cost while achieving superior average performance across languages. We note that, although we focus on the four languages to demonstrate the two-player data synthesis framework, DuoGuard retains its base model Qwen-2.5’s capacity to support all 29 languages.
• Figure 3: Overview of the two-player training pipeline. The generator produces synthetic data from seed data. The classifier makes predictions and we measure these examples as being predicted correctly or incorrectly based on their seed data label. We train the generator with DPO to create increasingly challenging examples, which in turn improve the classifier through iterative training.
• Figure 4: Relative performance decline (average F1 across six benchmarks and three languages) of various models compared to the English performance of DuoGuard.
• Figure 5: The F1 score on OpenAI benchmark of models trained with data containing different languages in our seed data. The inclusion of French in addition to English improves model performance on Spanish (36.9% to 62.8%) and German (31.9 to 59.6).

Theorems & Definitions (11)

• Theorem 4.1
• Proposition 1.1: Proposition 4 in azar2024general
• Lemma 1.2
• Lemma 1.3
• Lemma 1.4
• Theorem 1.5: Von Neumann's Minimax Theorem
• Theorem 1.6: Brouwer's Fixed Point Theorem
• Theorem 1.7: Banach Fixed Point Theorem
• Example 4.1
• Example 4.2