A Survey of Large Language Models for European Languages
Wazir Ali, Sampo Pyysalo
TL;DR
This survey addresses the gap in systematic coverage of large language models for official European languages by detailing the landscape of transformer-based architectures (encoder-only, decoder-only, encoder-decoder, and MoE), the pretraining datasets that enable EU-language modeling, and the spectrum of monolingual and multilingual LLMs tailored to European languages. It highlights data resources such as OSCAR, mC4, ROOTS, OpenSubtitles, and CULTURAX, and documents language-specific models (e.g., German GottBERT, French CamemBERT, Italian Fauno) alongside multilingual efforts (CroissantLLM, Viking, AURORA-M). The paper argues that this is the first comprehensive review focused on EU languages, providing a valuable inventory for researchers and practitioners to navigate data availability, model architectures, and cross-language transfer opportunities in the European context. Overall, it serves as a foundational reference for advancing resource-aware LLM development across high-, mid-, and low-resource European languages, guiding future data collection, benchmark creation, and model evaluation.
Abstract
Large Language Models (LLMs) have gained significant attention due to their high performance on a wide range of natural language tasks since the release of ChatGPT. The LLMs learn to understand and generate language by training billions of model parameters on vast volumes of text data. Despite being a relatively new field, LLM research is rapidly advancing in various directions. In this paper, we present an overview of LLM families, including LLaMA, PaLM, GPT, and MoE, and the methods developed to create and enhance LLMs for official European Union (EU) languages. We provide a comprehensive summary of common monolingual and multilingual datasets used for pretraining large language models.