Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Orion-14B: Open-source Multilingual Large Language Models

Du Chen, Yi Huang, Xiaopu Li, Yongqiang Li, Yongqiang Liu, Haihui Pan, Leichao Xu, Dacheng Zhang, Zhipeng Zhang, Kun Han

TL;DR

Orion-14B presents a multilingual 14B-parameter LLM trained on 2.5T tokens using a data-scheduling curriculum with staged language and content complexity. It provides a base model and Chat-oriented fine-tuned variants, and evaluates them across standard, multilingual, and retrieval-augmented tasks, reporting state-of-the-art results in several benchmarks. The work details data collection, quality filtering, deduplication, tokenizer choices, architectural tweaks, and a 2.5T-tokens pretraining regime, plus supervised fine-tuning and evaluation methodologies. It also introduces extension models (long-context, quantized, RAG, plugins) and openly releases code and models to foster reproducibility and broader adoption.

Abstract

In this study, we introduce Orion-14B, a collection of multilingual large language models with 14 billion parameters. We utilize a data scheduling approach to train a foundational model on a diverse corpus of 2.5 trillion tokens, sourced from texts in English, Chinese, Japanese, Korean, and other languages. Additionally, we fine-tuned a series of models tailored for conversational applications and other specific use cases. Our evaluation results demonstrate that Orion-14B achieves state-of-the-art performance across a broad spectrum of tasks. We make the Orion-14B model family and its associated code publicly accessible https://github.com/OrionStarAI/Orion, aiming to inspire future research and practical applications in the field.

Orion-14B: Open-source Multilingual Large Language Models

TL;DR

Orion-14B presents a multilingual 14B-parameter LLM trained on 2.5T tokens using a data-scheduling curriculum with staged language and content complexity. It provides a base model and Chat-oriented fine-tuned variants, and evaluates them across standard, multilingual, and retrieval-augmented tasks, reporting state-of-the-art results in several benchmarks. The work details data collection, quality filtering, deduplication, tokenizer choices, architectural tweaks, and a 2.5T-tokens pretraining regime, plus supervised fine-tuning and evaluation methodologies. It also introduces extension models (long-context, quantized, RAG, plugins) and openly releases code and models to foster reproducibility and broader adoption.

Abstract

In this study, we introduce Orion-14B, a collection of multilingual large language models with 14 billion parameters. We utilize a data scheduling approach to train a foundational model on a diverse corpus of 2.5 trillion tokens, sourced from texts in English, Chinese, Japanese, Korean, and other languages. Additionally, we fine-tuned a series of models tailored for conversational applications and other specific use cases. Our evaluation results demonstrate that Orion-14B achieves state-of-the-art performance across a broad spectrum of tasks. We make the Orion-14B model family and its associated code publicly accessible https://github.com/OrionStarAI/Orion, aiming to inspire future research and practical applications in the field.
Paper Structure (23 sections, 2 figures, 22 tables)

This paper contains 23 sections, 2 figures, 22 tables.

Table of Contents

  1. Introduction
  2. Data
  3. Data Source
  4. Data Quality
  5. Deduplication
  6. Pretraining
  7. Tokenizer
  8. Architecture
  9. Infrastructure
  10. Training Process
  11. Data Scheduling
  12. Fine-tuning
  13. SFT Data
  14. Training details
  15. Evaluation
  16. ...and 8 more sections

Figures (2)

  • Figure 1: Data sources distribution.
  • Figure 2: Validation loss during the training process. The validation set consists of 5,000 documents including a diverse collection of English and Chinese texts sourced from a variety of data sources.