Aligning LLMs for Multilingual Consistency in Enterprise Applications

TL;DR

The paper tackles multilingual inconsistencies in enterprise-enabled LLMs, driven by English-centric pretraining, and introduces a batch-wise alignment method that uses semantically equivalent multilingual data within each training batch to align internal reasoning and generation across languages. By combining Direct Preference Optimization and Odds-Ratio Preference Optimization within a batch-composition framework, the approach achieves up to 23.9% non-English accuracy gains without English degradation, and generalizes to unseen languages and out-of-domain tasks. Evaluations are conducted in a controlled RAG setup with six non-English languages plus English, demonstrating improvements in reasoning, fluency, and semantic understanding across benchmarks like MMMLU, MGSM, LAMBADA, and PAWS-X. The method is simple to implement, scalable, and integrates with existing training pipelines, offering a practical path toward more reliable and equitable multilingual AI in industry.

Abstract

Large language models (LLMs) remain unreliable for global enterprise applications due to substantial performance gaps between high-resource and mid/low-resource languages, driven by English-centric pretraining and internal reasoning biases. This inconsistency undermines customer experience and operational reliability in multilingual settings such as customer support, content moderation, and information retrieval. Even with advanced Retrieval-Augmented Generation (RAG) systems, we observe up to an 29% accuracy drop in non-English languages compared to English. We propose a practical, batch-wise alignment strategy for fine-tuning LLMs, leveraging semantically equivalent multilingual data in each training batch to directly align model outputs across languages. This approach improves non-English accuracy by up to 23.9% without compromising English performance, model reasoning, or retrieval quality. Our method is simple to implement, scalable, and integrates seamlessly with existing LLM training & deployment pipelines, enabling more robust and equitable multilingual AI solutions in industry.

Figures (7)

• Figure 1: Highlights the training paradigm with the proposed batch-composition technique for a batch with Positive Samples. The batch consists of training samples, with the same Query-Document composition across English and non-English to maintain same semantic consistency during training, where the LLM is expected to respond; "Yes, answer is present" or "I don't know" for the training samples.
• Figure 2: Highlights the high-level data-pipeline used to create the dataset. Each query is mapped by the Dataset Constructor to create a positive & negative sample with corresponding documents of the respective language.
• Figure 3: LLM (instruct versions) accuracy in English (left) and averaged over six non-English languages (right) for Baseline, Default ORPO (uncoupled), and ORPO with batch alignment (Ours). Batch alignment substantially improves non-English accuracy and cross-lingual consistency. See Table \ref{['tab3']} for full results.
• Figure 4: Impact of including vs. excluding English in training batches on non-English accuracy. Across all models, training with English (purple and green bars) consistently leads to higher non-English accuracy, demonstrating the role of high-resource languages in improving multilingual consistency. We use the instruct version of each model.
• Figure 5: Illustrates avg. performance when machine translated data is used for batch-aligned finetuning of models across Thai, Vietnamese, Hungarian & Romanian langauge.