Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bielik 11B v2 Technical Report

Krzysztof Ociepa, Łukasz Flis, Krzysztof Wróbel, Adrian Gwoździej, Remigiusz Kinas

TL;DR

The Bielik 11B v2 technical report presents a Polish-optimized Transformer model built on depth-upscaled Mistral 7B v0.2, achieving state-of-the-art performance for its size. It introduces Weighted Instruction Cross-Entropy Loss and Adaptive Learning Rate, trains on a massive Polish corpus, and leverages a comprehensive post-training regime including SFT and RLHF with DPO-P. Across Open PL LLM Leaderboard, MT-Bench, EQ-Bench, CPTUB, PLCC, and other benchmarks, Bielik demonstrates strong Polish-language capabilities with robust cross-lingual transfer and exceptional parameter efficiency, aided by quantization options for deployment. Practical impact includes enabling high-quality Polish NLP on consumer hardware while reducing resource demands; future work will extend domain specialization, cross-Slavic transfer, and enhanced function-calling. Overall, Bielik 11B v2 advances accessible, efficient Polish language AI with competitive performance against substantially larger models.

Abstract

We present Bielik 11B v2, a state-of-the-art language model optimized for Polish text processing. Built on the Mistral 7B v0.2 architecture and scaled to 11B parameters using depth up-scaling, this model demonstrates exceptional performance across Polish language benchmarks while maintaining strong cross-lingual capabilities. We introduce two key technical innovations: Weighted Instruction Cross-Entropy Loss, which optimizes learning across diverse instruction types by assigning quality-based weights to training examples, and Adaptive Learning Rate, which dynamically adjusts based on context length. Comprehensive evaluation across multiple benchmarks demonstrates that Bielik 11B v2 outperforms many larger models, including those with 2-6 times more parameters, and significantly surpasses other specialized Polish language models on tasks ranging from linguistic understanding to complex reasoning. The model's parameter efficiency and extensive quantization options enable deployment across various hardware configurations, advancing Polish language AI capabilities and establishing new benchmarks for resource-efficient language modeling in less-represented languages.

Bielik 11B v2 Technical Report

TL;DR

The Bielik 11B v2 technical report presents a Polish-optimized Transformer model built on depth-upscaled Mistral 7B v0.2, achieving state-of-the-art performance for its size. It introduces Weighted Instruction Cross-Entropy Loss and Adaptive Learning Rate, trains on a massive Polish corpus, and leverages a comprehensive post-training regime including SFT and RLHF with DPO-P. Across Open PL LLM Leaderboard, MT-Bench, EQ-Bench, CPTUB, PLCC, and other benchmarks, Bielik demonstrates strong Polish-language capabilities with robust cross-lingual transfer and exceptional parameter efficiency, aided by quantization options for deployment. Practical impact includes enabling high-quality Polish NLP on consumer hardware while reducing resource demands; future work will extend domain specialization, cross-Slavic transfer, and enhanced function-calling. Overall, Bielik 11B v2 advances accessible, efficient Polish language AI with competitive performance against substantially larger models.

Abstract

We present Bielik 11B v2, a state-of-the-art language model optimized for Polish text processing. Built on the Mistral 7B v0.2 architecture and scaled to 11B parameters using depth up-scaling, this model demonstrates exceptional performance across Polish language benchmarks while maintaining strong cross-lingual capabilities. We introduce two key technical innovations: Weighted Instruction Cross-Entropy Loss, which optimizes learning across diverse instruction types by assigning quality-based weights to training examples, and Adaptive Learning Rate, which dynamically adjusts based on context length. Comprehensive evaluation across multiple benchmarks demonstrates that Bielik 11B v2 outperforms many larger models, including those with 2-6 times more parameters, and significantly surpasses other specialized Polish language models on tasks ranging from linguistic understanding to complex reasoning. The model's parameter efficiency and extensive quantization options enable deployment across various hardware configurations, advancing Polish language AI capabilities and establishing new benchmarks for resource-efficient language modeling in less-represented languages.
Paper Structure (74 sections, 5 equations, 6 figures, 28 tables)

This paper contains 74 sections, 5 equations, 6 figures, 28 tables.

Table of Contents

  1. Introduction
  2. Architecture
  3. Pre-training
  4. Pre-training Data
  5. Data Cleanup
  6. Quality Evaluation
  7. Alternative high-capacity model.
  8. Category Classification: Results and Applications
  9. Training Hyperparameters
  10. Training process monitoring
  11. Post-training
  12. Post-training Data
  13. Supervised Fine-Tuning
  14. Masked Tokens
  15. Adaptive Learning Rate
  16. ...and 59 more sections

Figures (6)

  • Figure 1: Depth up-scaling with $n = 32$, $m = 7$, and $s = 50$.
  • Figure 2: Confusion matrix showing test and validation results for the XGBoost classifier.
  • Figure 3: Distribution of major thematic categories in the Polish text dataset ($\geq1.7\%$)
  • Figure 4: Distribution of instruction categories in the SFT dataset.
  • Figure 5: Weighted distribution of instruction categories in the SFT dataset.
  • ...and 1 more figures