Metadata Conditioned Large Language Models for Localization

TL;DR

Metadata conditioning localizes large language models by prefixing training data with URL, country, and continent tags, enabling region-specific knowledge without sacrificing global generalization. Across four controlled experiments, local models benefit from metadata during training, global models can be effectively localized via inference-time metadata (with URL metadata proving especially informative), and URL-only conditioning often suffices to capture geographic signals. Ablations show that while URL metadata is powerful, balanced data from all regions remains essential for robust performance, and metadata cannot fully compensate for missing regions. A downstream benchmark with 800 localized news MCQs demonstrates that metadata-conditioned global models can match the accuracy of substantially larger, instruction-tuned baselines, underscoring the approach's compute efficiency and practical impact for globally competent yet locally aware language systems.

Abstract

Large language models are typically trained by treating text as a single global distribution, often resulting in geographically homogenized behavior. We study metadata conditioning as a lightweight approach for localization, pre-training 31 models (at 0.5B and 1B parameter scales) from scratch on large-scale English news data annotated with verified URLs, country tags, and continent tags, covering 4 continents and 17 countries. Across four controlled experiments, we show that metadata conditioning consistently improves in-region performance without sacrificing cross-region generalization, enables global models to recover localization comparable to region-specific models, and improves learning efficiency. Our ablation studies demonstrate that URL-level metadata alone captures much of the geographic signal, while balanced regional data coverage remains essential, as metadata cannot fully compensate for missing regions. Finally, we introduce a downstream benchmark of 800 localized news MCQs and show that after instruction tuning, metadata conditioned global models achieve accuracy comparable to LLaMA-3.2-1B-Instruct, despite being trained on substantially less data. Together, these results establish metadata conditioning as a practical and compute-efficient approach for localization of language models.

Figures (20)

• Figure 1: We pre-train LLMs with metadata augmented news articles, and evaluate them to determine the effect of metadata conditioning on localization.
• Figure 2: Example of a training document formatted with metadata conditioning. Metadata fields are prepended to the document content.
• Figure 3: models have higher perplexities on cross-continent test sets (off-diagonal) than on local ones. Positive differences between perplexities of models trained without metadata and with metadata on the same test sets indicate the effectiveness of metadata in improving cross-region generalization while maintaining local performance.
• Figure 4: models conditioned with metadata during training and inference have lower perplexities on local test sets than the control model which is not conditioned on any metadata.
• Figure 5: models conditioned with metadata during training and inference have lower perplexities than the control variant , and have similar perplexities to the models on local test sets.