Beyond English: The Impact of Prompt Translation Strategies across Languages and Tasks in Multilingual LLMs

TL;DR

The paper investigates selective pre-translation prompting in multilingual LLMs by formalizing prompts into four components and evaluating all translation configurations across 35 languages and four tasks. It demonstrates that selective pre-translation consistently outperforms full English pre-translation and direct inference, with the largest gains for low-resource languages. The study analyzes how context, examples, and output language interact, deriving practical guidelines and a rule-based approach to selecting configurations. It further examines translation quality, showing that selective pre-translation can mitigate translation weaknesses, though translation quality remains a factor; limitations include a subset of LLMs and adherence challenges. Overall, the work provides a systematic, generalizable framework for leveraging selective pre-translation in real-world multilingual LLM applications.

Abstract

Despite advances in the multilingual capabilities of Large Language Models (LLMs) across diverse tasks, English remains the dominant language for LLM research and development. So, when working with a different language, this has led to the widespread practice of pre-translation, i.e., translating the task prompt into English before inference. Selective pre-translation, a more surgical approach, focuses on translating specific prompt components. However, its current use is sporagic and lacks a systematic research foundation. Consequently, the optimal pre-translation strategy for various multilingual settings and tasks remains unclear. In this work, we aim to uncover the optimal setup for pre-translation by systematically assessing its use. Specifically, we view the prompt as a modular entity, composed of four functional parts: instruction, context, examples, and output, either of which could be translated or not. We evaluate pre-translation strategies across 35 languages covering both low and high-resource languages, on various tasks including Question Answering (QA), Natural Language Inference (NLI), Named Entity Recognition (NER), and Abstractive Summarization. Our experiments show the impact of factors as similarity to English, translation quality and the size of pre-trained data, on the model performance with pre-translation. We suggest practical guidelines for choosing optimal strategies in various multilingual settings.

Figures (12)

• Figure 1: Prompting Strategies: Direct Inference, Selective Pre-Translation, and Pre-Translation
• Figure 2: Performance Gap Analysis for the Examples language (English minus Source). Left to right X-axes order indicates Low to High Resource Level. Y-axes indicate language preference: positive values for the source language, and negative for the English language.
• Figure 3: Scatter plot showing the relationship between syntactic similarity to English (further right is more similar) and translation quality (ROUGE) for four language resource subsets (represented as distinct four colored shapes). Each dot represents a different language. Positive linear regression shows an upward trend.
• Figure 4: Correlation between translation quality (BERTScore) and accuracy (F1) for Pre-Translation-Zero-shot prompting, each dot is a different language.
• Figure 5: Examples of 3 configurations of German. Each configuration is in the following format <Instruction,Context,Examples,Output>