Bielik 11B v3: Multilingual Large Language Model for European Languages

TL;DR

Bielik 11B v3 presents an $11$-billion-parameter multilingual LLM optimized for Polish, built by depth up-scaling the Mistral 7B v0.2 base to $s=50$ layers and trained via a four-stage pipeline (continuous pre-training, supervised fine-tuning, Direct Preference Optimization with DPO-P, and reinforcement learning). It leverages a massively expanded multilingual corpus spanning $32$ languages and over $1.1$ trillion tokens, with careful data processing, ethical controls, and long-context capabilities ($8{,}192$, $32{,}768$, and $65{,}536$ token contexts across stages). The model demonstrates state-of-the-art performance on Polish-specific benchmarks (e.g., PLCC, Open PL LLM Leaderboard, Belebele) while maintaining competitive English and multilingual capabilities, often outperforming larger models by significant margins and showing strong translation and reasoning abilities (FLORES, CPTUB). Its design emphasizes resource efficiency and deployment across hardware with varying constraints, highlighting practical impact for underrepresented European languages. The work also outlines future directions toward larger variants, broader language coverage, domain adaptations, and potential multimodal extensions.

Abstract

We present Bielik 11B v3, a state-of-the-art language model highly optimized for the Polish language, while also maintaining strong capabilities in other European languages. This model extends the Mistral 7B v0.2 architecture, scaled to 11B parameters via depth up-scaling. Its development involved a comprehensive four-stage training pipeline: continuous pre-training, supervised fine-tuning (SFT), Direct Preference Optimization (DPO), and reinforcement learning. Comprehensive evaluations demonstrate that Bielik 11B v3 achieves exceptional performance. It significantly surpasses other specialized Polish language models and outperforms many larger models (with 2-6 times more parameters) on a wide range of tasks, from basic linguistic understanding to complex reasoning. The model's parameter efficiency, combined with extensive quantization options, allows for effective deployment across diverse hardware configurations. Bielik 11B v3 not only advances AI capabilities for the Polish language but also establishes a new benchmark for developing resource-efficient, high-performance models for less-represented languages.

Figures (2)

• Figure 1: Depth up-scaling with $n = 32$, $m = 7$, and $s = 50$.
• Figure 2: Average score of the Bielik 11B v3 model checkpoints during pretraining in the Open PL LLM benchmark