Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Art of Asking: Multilingual Prompt Optimization for Synthetic Data

David Mora, Viraat Aryabumi, Wei-Yin Ko, Sara Hooker, Julia Kreutzer, Marzieh Fadaee

TL;DR

The paper addresses multilingual synthetic data generation by shifting the input prompt distribution rather than solely maximizing the quality of generated completions. It introduces a prompt-space optimization framework with three transformations—Naturalness, Cultural Adaptation, and Difficulty Enhancement—applied via a strong teacher LLM across 12 languages. Empirical results show consistent downstream gains across a broad set of benchmarks, including notable improvements on open-ended tasks and under-resourced languages, underscoring the value of culturally grounded, linguistically rich prompts. The work argues that optimizing prompts yields more robust, globally capable LLMs and lays a foundation for further refinements in synthetic data quality and multilingual evaluation.

Abstract

Synthetic data has become a cornerstone for scaling large language models, yet its multilingual use remains bottlenecked by translation-based prompts. This strategy inherits English-centric framing and style and neglects cultural dimensions, ultimately constraining model generalization. We argue that the overlooked prompt space-the very inputs that define training distributions-offers a more powerful lever for improving multilingual performance. We introduce a lightweight framework for prompt-space optimization, where translated prompts are systematically transformed for Naturalness, Cultural Adaptation, and Difficulty Enhancement. Using an off-the-shelf multilingual LLM, we apply these transformations to prompts for 12 languages spanning 7 families. Under identical data conditions, our approaches achieve substantial and consistent downstream improvements over the translation-only baseline: +4.7% on Global-MMLU accuracy, +2.4% on Flores XCometXL and +35.3% wins in preferences on mArenaHard. We establish prompt-space optimization as a simple yet powerful paradigm for building multilingual LLMs that are more robust, culturally grounded, and globally capable.

The Art of Asking: Multilingual Prompt Optimization for Synthetic Data

TL;DR

The paper addresses multilingual synthetic data generation by shifting the input prompt distribution rather than solely maximizing the quality of generated completions. It introduces a prompt-space optimization framework with three transformations—Naturalness, Cultural Adaptation, and Difficulty Enhancement—applied via a strong teacher LLM across 12 languages. Empirical results show consistent downstream gains across a broad set of benchmarks, including notable improvements on open-ended tasks and under-resourced languages, underscoring the value of culturally grounded, linguistically rich prompts. The work argues that optimizing prompts yields more robust, globally capable LLMs and lays a foundation for further refinements in synthetic data quality and multilingual evaluation.

Abstract

Synthetic data has become a cornerstone for scaling large language models, yet its multilingual use remains bottlenecked by translation-based prompts. This strategy inherits English-centric framing and style and neglects cultural dimensions, ultimately constraining model generalization. We argue that the overlooked prompt space-the very inputs that define training distributions-offers a more powerful lever for improving multilingual performance. We introduce a lightweight framework for prompt-space optimization, where translated prompts are systematically transformed for Naturalness, Cultural Adaptation, and Difficulty Enhancement. Using an off-the-shelf multilingual LLM, we apply these transformations to prompts for 12 languages spanning 7 families. Under identical data conditions, our approaches achieve substantial and consistent downstream improvements over the translation-only baseline: +4.7% on Global-MMLU accuracy, +2.4% on Flores XCometXL and +35.3% wins in preferences on mArenaHard. We establish prompt-space optimization as a simple yet powerful paradigm for building multilingual LLMs that are more robust, culturally grounded, and globally capable.
Paper Structure (52 sections, 3 equations, 4 figures, 23 tables)

This paper contains 52 sections, 3 equations, 4 figures, 23 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Problem Setup
  4. Transformation Operators
  5. Prompt Tuning
  6. Experiments
  7. Data Processing Pipeline
  8. English Seed Prompts
  9. Prompt Translation into Target Languages
  10. Prompt Optimization
  11. Prompt Completions
  12. Fine-Tuning
  13. Base Model
  14. Data Mixture
  15. Evaluation
  16. ...and 37 more sections

Figures (4)

  • Figure 1: Prompt transformations consistently improve over translations: Comparison of translated model and our most well-rounded method (Cultural+Difficulty Mix) across different multilingual benchmarks. mArenaHard and Polywrite win-rates are in direct comparison between the two models.
  • Figure 2: Illustration of our prompt transformations on a representative toy example that gets adapted for German: Each transformation modifies the original English prompt, with major modifications highlighted in bold. Modifications to the prompt cause changes in the generation as well, so by making the prompt more natural by using the German term "Erntedankfest" rather than the English "Thanksgiving", the completion now lists typical German rather than American Thanksgiving dishes ("bread, fruit, pumpkin, potato dishes"). The Cultural Adaptationfurther localizes the prompt ("in Germany") and replaces the event of Thanksgiving with the event of Christmas, which has larger significance in German culture. The Difficulty transformation yields a prompt that requests a shopping list for side dishes of Thanksgiving, making it more specific but also more complex. Full examples of prompt transformations and their corresponding completions that were used for our experiments are in Table \ref{['app-tab:examples-table']}.
  • Figure 3: Translation performance on Flores by language (grouped by those supported in pretraining vs others), compared also against the teacher model.
  • Figure 4: Performance on lowest-resource languages Welsh (cy), Basque (eu) and Latvian (lv) across three tasks. Win Rates are in comparison with the Translated baseline.