Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NorwAI's Large Language Models: Technical Report

Jon Atle Gulla, Peng Liu, Lemei Zhang

TL;DR

NorwAI presents a comprehensive suite of Norwegian-focused large language models spanning multiple architectures and scales, with extensive pretraining on a large, progressively expanded Norwegian corpus and targeted post-training strategies. The work combines instruction tuning, RLHF-based preference optimization, and efficient deployment practices to produce robust, domain-adapted models that perform well on Norwegian tasks and real-world benchmarks like NLEBench and NorEval. Key contributions include a diverse model catalog, a Norwegian data pipeline with publicly released corpora, a novel evaluation suite, and practical guidance for model selection, deployment, and domain adaptation in Nordic contexts. The work advances digital sovereignty and accessibility for Norwegian and wider Scandinavian language processing, enabling effects in education, media, healthcare, and government services.

Abstract

Norwegian, spoken by approximately five million people, remains underrepresented in many of the most significant breakthroughs in Natural Language Processing (NLP). To address this gap, the NorLLM team at NorwAI has developed a family of models specifically tailored to Norwegian and other Scandinavian languages, building on diverse Transformer-based architectures such as GPT, Mistral, Llama2, Mixtral and Magistral. These models are either pretrained from scratch or continually pretrained on 25B - 88.45B tokens, using a Norwegian-extended tokenizer and advanced post-training strategies to optimize performance, enhance robustness, and improve adaptability across various real-world tasks. Notably, instruction-tuned variants (e.g., Mistral-7B-Instruct and Mixtral-8x7B-Instruct) showcase strong assistant-style capabilities, underscoring their potential for practical deployment in interactive and domain-specific applications. The NorwAI large language models are openly available to Nordic organizations, companies and students for both research and experimental use. This report provides detailed documentation of the model architectures, training data, tokenizer design, fine-tuning strategies, deployment, and evaluations.

NorwAI's Large Language Models: Technical Report

TL;DR

NorwAI presents a comprehensive suite of Norwegian-focused large language models spanning multiple architectures and scales, with extensive pretraining on a large, progressively expanded Norwegian corpus and targeted post-training strategies. The work combines instruction tuning, RLHF-based preference optimization, and efficient deployment practices to produce robust, domain-adapted models that perform well on Norwegian tasks and real-world benchmarks like NLEBench and NorEval. Key contributions include a diverse model catalog, a Norwegian data pipeline with publicly released corpora, a novel evaluation suite, and practical guidance for model selection, deployment, and domain adaptation in Nordic contexts. The work advances digital sovereignty and accessibility for Norwegian and wider Scandinavian language processing, enabling effects in education, media, healthcare, and government services.

Abstract

Norwegian, spoken by approximately five million people, remains underrepresented in many of the most significant breakthroughs in Natural Language Processing (NLP). To address this gap, the NorLLM team at NorwAI has developed a family of models specifically tailored to Norwegian and other Scandinavian languages, building on diverse Transformer-based architectures such as GPT, Mistral, Llama2, Mixtral and Magistral. These models are either pretrained from scratch or continually pretrained on 25B - 88.45B tokens, using a Norwegian-extended tokenizer and advanced post-training strategies to optimize performance, enhance robustness, and improve adaptability across various real-world tasks. Notably, instruction-tuned variants (e.g., Mistral-7B-Instruct and Mixtral-8x7B-Instruct) showcase strong assistant-style capabilities, underscoring their potential for practical deployment in interactive and domain-specific applications. The NorwAI large language models are openly available to Nordic organizations, companies and students for both research and experimental use. This report provides detailed documentation of the model architectures, training data, tokenizer design, fine-tuning strategies, deployment, and evaluations.
Paper Structure (29 sections, 2 figures, 7 tables)

This paper contains 29 sections, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Available Models
  3. Model Training
  4. Model Configurations
  5. Pre-training
  6. Post-training
  7. Instruction Tuning and Alignment
  8. Preference Optimization
  9. Deployment Inference
  10. Transformers Example
  11. Quantization
  12. Alternative Hosting and Deployment Options
  13. Evaluation and Benchmarking
  14. Norwegian Benchmark Dataset - NLEBench
  15. Quantitative analysis
  16. ...and 14 more sections

Figures (2)

  • Figure 1: Statistics of the NorLLM training corpora, with the number of tokens measured in billions (A) and millions (B).
  • Figure 2: Human evaluation results on Schibsted news summarization.