Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bielik v3 Small: Technical Report

Krzysztof Ociepa, Łukasz Flis, Remigiusz Kinas, Krzysztof Wróbel, Adrian Gwoździej

TL;DR

Bielik v3 targets parameter-efficient Polish language modeling by combining depth-up-scaling of a Qwen2.5 backbone with a Polish-specific APT4 tokenizer, adaptive learning rate, and targeted data curation. The approach is complemented by synthetic data generation, quality-controlled training corpora, and preference-based RLHF (notably DPO-P) to align outputs with human preferences, achieving strong results on Polish benchmarks while staying resource-conscious. The 4.5B variant often matches or surpasses much larger models across benchmarks such as CPTUB, EQ-Bench, and medical reasoning datasets, illustrating substantial efficiency gains for less-resourced languages. The work demonstrates practical impact for deploying high-quality Polish NLP in constrained environments and establishes a benchmark for parameter-efficient language modeling in linguistically underrepresented contexts; future work will push deeper reasoning and broader domain coverage while maintaining efficiency.

Abstract

We introduce Bielik v3, a series of parameter-efficient generative text models (1.5B and 4.5B) optimized for Polish language processing. These models demonstrate that smaller, well-optimized architectures can achieve performance comparable to much larger counterparts while requiring substantially fewer computational resources. Our approach incorporates several key innovations: a custom Polish tokenizer (APT4) that significantly improves token efficiency, Weighted Instruction Cross-Entropy Loss to balance learning across instruction types, and Adaptive Learning Rate that dynamically adjusts based on training progress. Trained on a meticulously curated corpus of 292 billion tokens spanning 303 million documents, these models excel across multiple benchmarks, including the Open PL LLM Leaderboard, Complex Polish Text Understanding Benchmark, Polish EQ-Bench, and Polish Medical Leaderboard. The 4.5B parameter model achieves results competitive with models 2-3 times its size, while the 1.5B model delivers strong performance despite its extremely compact profile. These advances establish new benchmarks for parameter-efficient language modeling in less-represented languages, making high-quality Polish language AI more accessible for resource-constrained applications.

Bielik v3 Small: Technical Report

TL;DR

Bielik v3 targets parameter-efficient Polish language modeling by combining depth-up-scaling of a Qwen2.5 backbone with a Polish-specific APT4 tokenizer, adaptive learning rate, and targeted data curation. The approach is complemented by synthetic data generation, quality-controlled training corpora, and preference-based RLHF (notably DPO-P) to align outputs with human preferences, achieving strong results on Polish benchmarks while staying resource-conscious. The 4.5B variant often matches or surpasses much larger models across benchmarks such as CPTUB, EQ-Bench, and medical reasoning datasets, illustrating substantial efficiency gains for less-resourced languages. The work demonstrates practical impact for deploying high-quality Polish NLP in constrained environments and establishes a benchmark for parameter-efficient language modeling in linguistically underrepresented contexts; future work will push deeper reasoning and broader domain coverage while maintaining efficiency.

Abstract

We introduce Bielik v3, a series of parameter-efficient generative text models (1.5B and 4.5B) optimized for Polish language processing. These models demonstrate that smaller, well-optimized architectures can achieve performance comparable to much larger counterparts while requiring substantially fewer computational resources. Our approach incorporates several key innovations: a custom Polish tokenizer (APT4) that significantly improves token efficiency, Weighted Instruction Cross-Entropy Loss to balance learning across instruction types, and Adaptive Learning Rate that dynamically adjusts based on training progress. Trained on a meticulously curated corpus of 292 billion tokens spanning 303 million documents, these models excel across multiple benchmarks, including the Open PL LLM Leaderboard, Complex Polish Text Understanding Benchmark, Polish EQ-Bench, and Polish Medical Leaderboard. The 4.5B parameter model achieves results competitive with models 2-3 times its size, while the 1.5B model delivers strong performance despite its extremely compact profile. These advances establish new benchmarks for parameter-efficient language modeling in less-represented languages, making high-quality Polish language AI more accessible for resource-constrained applications.
Paper Structure (44 sections, 9 equations, 3 figures, 18 tables)

This paper contains 44 sections, 9 equations, 3 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Model and Tokenizer
  3. Model Architecture
  4. Tokenizer
  5. Pre-training
  6. Pre-training Data
  7. Data Cleanup
  8. Quality Evaluation
  9. Best model configuration.
  10. Category Classification: Results and Applications
  11. Synthetic Data Generation for Instruction Tuning
  12. Generation of High-Quality, Thematically Balanced QA Instructions
  13. Data Recycling: Improving Imperfect Texts for Inclusion
  14. Summary of the Synthetic Data Curation Strategy
  15. Post-training
  16. ...and 29 more sections

Figures (3)

  • Figure 1: Bielik 4.5B v3 model upscaling via Depth Up-Scaling ($n = 36$, $m = 8$, $s = 56$) with tokenizer replacement and outermost layer duplication.
  • Figure 2: Confusion matrix showing test and validation results for the XGBoost classifier.
  • Figure 3: Distribution of major thematic categories in the Polish text dataset ($\geq0.9\%$)